[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:06 dist-test-slave-8z53 krb5kdc[2998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:06 dist-test-slave-8z53 krb5kdc[2998](info): set up 2 sockets
Jun 20 20:18:06 dist-test-slave-8z53 krb5kdc[2998](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 20:18:07.225183 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:07.326453 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41795
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:07.424590 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:07.424820 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:07.424854 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:07.428601 3011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:07.428644 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:07.428665 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:07.428679 3011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:07.428692 3011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:07.433113 3011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:07.434010 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:07.434913 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:07.442898 3017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:07.442873 3019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:07.442920 3016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:07.442996 3011 server_base.cc:1034] running on GCE node
I20260620 20:18:07.443627 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:07.444629 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:07.445809 3011 hybrid_clock.cc:648] HybridClock initialized: now 1781986687445797 us; error 24 us; skew 500 ppm
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:07.448413 3011 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:07.449239 3011 webserver.cc:460] Webserver started at http://127.2.235.254:35585/ using document root <none> and password file <none>
I20260620 20:18:07.449805 3011 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:07.449860 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:07.450012 3011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:07.451831 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "abab9d1f68a14346b807e595a068cdf8"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "6e77b1ec852b454480de246ef4c074e1"
server_key_iv: "67a51bd3bf3976b0b487cbe9fce6e92e"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.452319 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "abab9d1f68a14346b807e595a068cdf8"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "6e77b1ec852b454480de246ef4c074e1"
server_key_iv: "67a51bd3bf3976b0b487cbe9fce6e92e"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.455751 3011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:18:07.457866 3026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.458943 3011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:07.459059 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "abab9d1f68a14346b807e595a068cdf8"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "6e77b1ec852b454480de246ef4c074e1"
server_key_iv: "67a51bd3bf3976b0b487cbe9fce6e92e"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.459147 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:07.478440 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:07.481060 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:07.481315 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:07.488432 3011 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:41795
I20260620 20:18:07.488446 3078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:41795 every 8 connection(s)
I20260620 20:18:07.489392 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:07.490995 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3011
I20260620 20:18:07.491149 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:07.492419 3079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:07.492630 2991 external_mini_cluster.cc:1428] Setting key 445d9bc6af016f6eaaf40e44deea5ecb
I20260620 20:18:07.498358 3079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8: Bootstrap starting.
I20260620 20:18:07.500403 3079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:07.501240 3079 log.cc:824] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:07.502980 3079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8: No bootstrap required, opened a new log
I20260620 20:18:07.506810 3079 raft_consensus.cc:348] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab9d1f68a14346b807e595a068cdf8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41795 } }
I20260620 20:18:07.507045 3079 raft_consensus.cc:374] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:07.507102 3079 raft_consensus.cc:729] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abab9d1f68a14346b807e595a068cdf8, State: Initialized, Role: FOLLOWER
I20260620 20:18:07.507584 3079 consensus_queue.cc:260] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab9d1f68a14346b807e595a068cdf8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41795 } }
I20260620 20:18:07.507700 3079 raft_consensus.cc:388] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:07.507776 3079 raft_consensus.cc:482] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:07.507887 3079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:07.508827 3079 raft_consensus.cc:504] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab9d1f68a14346b807e595a068cdf8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41795 } }
I20260620 20:18:07.509133 3079 leader_election.cc:302] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abab9d1f68a14346b807e595a068cdf8; no voters:
I20260620 20:18:07.509418 3079 leader_election.cc:288] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:07.509546 3084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:07.509768 3084 raft_consensus.cc:686] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [term 1 LEADER]: Becoming Leader. State: Replica: abab9d1f68a14346b807e595a068cdf8, State: Running, Role: LEADER
I20260620 20:18:07.510098 3084 consensus_queue.cc:237] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab9d1f68a14346b807e595a068cdf8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41795 } }
I20260620 20:18:07.510563 3079 sys_catalog.cc:564] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:07.511364 3084 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abab9d1f68a14346b807e595a068cdf8. Latest consensus state: current_term: 1 leader_uuid: "abab9d1f68a14346b807e595a068cdf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab9d1f68a14346b807e595a068cdf8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41795 } } }
I20260620 20:18:07.511492 3084 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:07.511915 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abab9d1f68a14346b807e595a068cdf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abab9d1f68a14346b807e595a068cdf8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41795 } } }
I20260620 20:18:07.512430 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:07.511914 3093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:07.515522 3093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:07.518241 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:07.496084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.2.235.254:41795)
0620 20:18:07.496526 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:07.496546 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:07.497601 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 20:18:07.500600 (+ 2999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:07.500625 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:07.500990 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:07.501365 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:07.503636 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.505125 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.506336 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.506972 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.507135 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:07.511742 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:07.511749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:07.511765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:07.511808 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:07.514495 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:07.515119 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:07.515125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:07.515131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:07.515271 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:07.515700 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:07.515704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:07.515706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:07.516176 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:07.516405 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:07.517013 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 20:18:07.517320 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260620 20:18:07.520579 3093 catalog_manager.cc:1347] Generated new cluster ID: 89c3794915bd422196fc83226408fe67
I20260620 20:18:07.520658 3093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:07.531723 3093 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:07.532500 3093 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:07.543061 3093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abab9d1f68a14346b807e595a068cdf8: Generated new TSK 0
I20260620 20:18:07.543588 3093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:07.574278 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41795
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:07.674368 3105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:07.674737 3105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:07.674813 3105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:07.678700 3105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:07.678745 3105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:07.678828 3105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:07.683534 3105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:07.684383 3105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:07.685333 3105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:07.693636 3110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:07.693686 3113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:07.693709 3111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:07.693717 3105 server_base.cc:1034] running on GCE node
I20260620 20:18:07.694334 3105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:07.694942 3105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:07.696101 3105 hybrid_clock.cc:648] HybridClock initialized: now 1781986687696087 us; error 24 us; skew 500 ppm
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:07.698660 3105 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:07.699365 3105 webserver.cc:460] Webserver started at http://127.2.235.193:44501/ using document root <none> and password file <none>
I20260620 20:18:07.699805 3105 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:07.699857 3105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:07.699996 3105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:07.701764 3105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "56aaac76fe52499f99c1687a454eb22f"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "202d33178447fcc81f769bd4f5ed956e"
server_key_iv: "629711f6a480a7767db584169ff84b7e"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.702250 3105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "56aaac76fe52499f99c1687a454eb22f"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "202d33178447fcc81f769bd4f5ed956e"
server_key_iv: "629711f6a480a7767db584169ff84b7e"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.705566 3105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:07.707458 3120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.708436 3105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:07.708568 3105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "56aaac76fe52499f99c1687a454eb22f"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "202d33178447fcc81f769bd4f5ed956e"
server_key_iv: "629711f6a480a7767db584169ff84b7e"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.708666 3105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:07.720319 3105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:07.722873 3105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:07.723068 3105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:07.723628 3105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:07.724390 3105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:07.724466 3105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.724532 3105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:07.724606 3105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.732882 3105 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:44711
I20260620 20:18:07.732926 3233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:44711 every 8 connection(s)
I20260620 20:18:07.733744 3105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:07.738902 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3105
I20260620 20:18:07.739040 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:07.739329 2991 external_mini_cluster.cc:1428] Setting key 0a07193dae6dd6e2355cb1fedfc7bf44
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:07.747421 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:07.735308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:59137 (local address 127.2.235.254:41795)
0620 20:18:07.735423 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:07.735426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:07.736095 (+ 669us) server_negotiation.cc:407] Connection header received
0620 20:18:07.737313 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:07.737316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:07.737371 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:07.737465 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:07.738458 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.738993 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.740434 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.740805 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.740879 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:07.743616 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:07.743623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:07.743629 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:07.743659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:07.745372 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:07.745837 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:07.745842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:07.745845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:07.745935 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:07.746246 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:07.746251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:07.746254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:07.746485 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:07.746592 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:07.747117 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 20:18:07.747253 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:18:07.748162 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:07.735603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41795 (local address 127.2.235.193:59137)
0620 20:18:07.735978 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:07.736012 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:07.737081 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:07.737608 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:07.737614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:07.737932 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:07.738301 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:07.738311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.739172 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.739177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:07.740321 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:07.740331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.740941 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.740946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:07.741147 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:07.741678 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:07.741703 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:07.743457 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:07.745474 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:07.745478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:07.745487 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:07.745735 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:07.746029 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:07.746031 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:07.746032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:07.746152 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:07.746585 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:07.746589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:07.746844 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 20:18:07.747023 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 20:18:07.747317 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:07.749361 3234 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41795
I20260620 20:18:07.749557 3234 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:07.749989 3234 heartbeater.cc:511] Master 127.2.235.254:41795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:18:07.751461 3043 ts_manager.cc:194] Registered new tserver with Master: 56aaac76fe52499f99c1687a454eb22f (127.2.235.193:44711)
I20260620 20:18:07.752460 3043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:59137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:07.763598 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41795
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:07.859220 3239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:07.859411 3239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:07.859481 3239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:07.863720 3239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:07.863778 3239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:07.863888 3239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:07.868830 3239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:07.869680 3239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:07.870529 3239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:07.879151 3244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:07.879223 3245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:07.879258 3247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:07.879382 3239 server_base.cc:1034] running on GCE node
I20260620 20:18:07.879781 3239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:07.880355 3239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:07.881544 3239 hybrid_clock.cc:648] HybridClock initialized: now 1781986687881532 us; error 22 us; skew 500 ppm
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:07.884244 3239 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:07.884931 3239 webserver.cc:460] Webserver started at http://127.2.235.194:45073/ using document root <none> and password file <none>
I20260620 20:18:07.885500 3239 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:07.885560 3239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:07.885710 3239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:07.887424 3239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "838bbfa25742408eb91a4b29fe5ee006"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "0f8ed8263940dbc124ba94958b05d362"
server_key_iv: "fba1f3f2e9fed5112c6746a7b0b27224"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.887933 3239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "838bbfa25742408eb91a4b29fe5ee006"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "0f8ed8263940dbc124ba94958b05d362"
server_key_iv: "fba1f3f2e9fed5112c6746a7b0b27224"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.891076 3239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:07.893033 3254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.894230 3239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:07.894367 3239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "838bbfa25742408eb91a4b29fe5ee006"
format_stamp: "Formatted at 2026-06-20 20:18:07 on dist-test-slave-8z53"
server_key: "0f8ed8263940dbc124ba94958b05d362"
server_key_iv: "fba1f3f2e9fed5112c6746a7b0b27224"
server_key_version: "encryptionkey@0"
I20260620 20:18:07.894476 3239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:07.907266 3239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:07.909793 3239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:07.909966 3239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:07.910526 3239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:07.911296 3239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:07.911368 3239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.911448 3239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:07.911495 3239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:07.919250 3239 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:41035
I20260620 20:18:07.919339 3367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:41035 every 8 connection(s)
I20260620 20:18:07.919993 3239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
Jun 20 20:18:07 dist-test-slave-8z53 krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986687, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:07.928212 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3239
I20260620 20:18:07.928315 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:07.928594 2991 external_mini_cluster.cc:1428] Setting key 25a4f20c136af1eb0e90bebfa12ff948
I20260620 20:18:07.931789 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:07.921498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:60193 (local address 127.2.235.254:41795)
0620 20:18:07.921626 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:07.921631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:07.922392 (+ 761us) server_negotiation.cc:407] Connection header received
0620 20:18:07.923586 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:07.923588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:07.923632 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:07.923711 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:07.924721 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.925174 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.926210 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.926450 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.926519 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:07.928545 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:07.928548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:07.928551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:07.928575 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:07.929930 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:07.930384 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:07.930387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:07.930389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:07.930466 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:07.930815 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:07.930819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:07.930820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:07.930988 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:07.931089 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:07.931517 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 20:18:07.931626 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:18:07.932543 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:07.921852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41795 (local address 127.2.235.194:60193)
0620 20:18:07.922258 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:07.922295 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:07.923384 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:07.923853 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:07.923859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:07.924156 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:07.924577 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:07.924589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.925326 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.925333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:07.926111 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:07.926118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:07.926555 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:07.926557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:07.926670 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:07.927086 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:07.927099 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:07.928398 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:07.930027 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:07.930031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:07.930042 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:07.930282 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:07.930571 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:07.930575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:07.930577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:07.930712 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:07.931087 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:07.931093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:07.931306 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 20:18:07.931475 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 20:18:07.931757 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260620 20:18:07.933643 3368 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41795
I20260620 20:18:07.933856 3368 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:07.934271 3368 heartbeater.cc:511] Master 127.2.235.254:41795 requested a full tablet report, sending...
I20260620 20:18:07.935226 3043 ts_manager.cc:194] Registered new tserver with Master: 838bbfa25742408eb91a4b29fe5ee006 (127.2.235.194:41035)
I20260620 20:18:07.935999 3043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:60193
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:07.953214 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41795
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:08.048878 3373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:08.049080 3373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:08.049144 3373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:08.052690 3373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:08.052758 3373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:08.052834 3373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:08.057406 3373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41795
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:08.058179 3373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:08.058945 3373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:08.067066 3378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:08.067080 3379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:08.067066 3381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:08.067442 3373 server_base.cc:1034] running on GCE node
I20260620 20:18:08.067785 3373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:08.068307 3373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:08.069470 3373 hybrid_clock.cc:648] HybridClock initialized: now 1781986688069458 us; error 30 us; skew 500 ppm
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:08.071838 3373 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:08.072594 3373 webserver.cc:460] Webserver started at http://127.2.235.195:43227/ using document root <none> and password file <none>
I20260620 20:18:08.073045 3373 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:08.073124 3373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:08.073388 3373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:08.075135 3373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "e372806f23b24bd08c4ca5b6eb8d01d6"
format_stamp: "Formatted at 2026-06-20 20:18:08 on dist-test-slave-8z53"
server_key: "41c28ed6f1e942dbd34eb12bd090efed"
server_key_iv: "ea7f0dabaa2d219b229c1806cec36dbe"
server_key_version: "encryptionkey@0"
I20260620 20:18:08.075678 3373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "e372806f23b24bd08c4ca5b6eb8d01d6"
format_stamp: "Formatted at 2026-06-20 20:18:08 on dist-test-slave-8z53"
server_key: "41c28ed6f1e942dbd34eb12bd090efed"
server_key_iv: "ea7f0dabaa2d219b229c1806cec36dbe"
server_key_version: "encryptionkey@0"
I20260620 20:18:08.078800 3373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:18:08.080754 3388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:08.081887 3373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:18:08.081965 3373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "e372806f23b24bd08c4ca5b6eb8d01d6"
format_stamp: "Formatted at 2026-06-20 20:18:08 on dist-test-slave-8z53"
server_key: "41c28ed6f1e942dbd34eb12bd090efed"
server_key_iv: "ea7f0dabaa2d219b229c1806cec36dbe"
server_key_version: "encryptionkey@0"
I20260620 20:18:08.082072 3373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:08.096513 3373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:08.099377 3373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:08.099551 3373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:08.100033 3373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:08.100724 3373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:08.100764 3373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:08.100821 3373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:08.100855 3373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:08.109153 3373 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:39799
I20260620 20:18:08.109176 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:39799 every 8 connection(s)
I20260620 20:18:08.109892 3373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:08.117827 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3373
I20260620 20:18:08.117933 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:08.118214 2991 external_mini_cluster.cc:1428] Setting key 6be8a4fcdbc368f1f9649b01fabac5c7
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:08.123086 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:08.111798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:56787 (local address 127.2.235.254:41795)
0620 20:18:08.111951 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:08.111954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:08.112661 (+ 707us) server_negotiation.cc:407] Connection header received
0620 20:18:08.114141 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:08.114145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:08.114207 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:08.114322 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:08.115343 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.115828 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.116838 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.117083 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.117202 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:08.119600 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:08.119607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:08.119609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:08.119646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:08.121337 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:08.121902 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:08.121905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:08.121906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:08.121988 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:08.122291 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:08.122293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:08.122294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:08.122444 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:08.122514 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:08.122885 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:18:08.122980 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:08.123776 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:08.112146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41795 (local address 127.2.235.195:56787)
0620 20:18:08.112518 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:08.112577 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:08.113927 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:08.114476 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:08.114492 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:08.114820 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:08.115205 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:08.115216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.115947 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.115951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:08.116714 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:08.116722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.117229 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.117234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:08.117416 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:08.117879 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:08.117902 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:08.119433 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:08.121468 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:08.121472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:08.121481 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:08.121715 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:08.122077 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:08.122079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:08.122080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:08.122213 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:08.122522 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:08.122525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:08.122698 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 20:18:08.122825 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 20:18:08.123011 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20260620 20:18:08.124846 3502 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41795
I20260620 20:18:08.125034 3502 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:08.125468 3502 heartbeater.cc:511] Master 127.2.235.254:41795 requested a full tablet report, sending...
I20260620 20:18:08.126339 3043 ts_manager.cc:194] Registered new tserver with Master: e372806f23b24bd08c4ca5b6eb8d01d6 (127.2.235.195:39799)
I20260620 20:18:08.126814 3043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:56787
I20260620 20:18:08.132189 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:08.154740 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:08.146548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.2.235.193:44711)
0620 20:18:08.146846 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:08.146855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:08.146873 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:18:08.146913 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:08.146916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:08.147044 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:08.147153 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:08.147669 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.148341 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.149029 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.149400 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.149476 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:08.151440 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:08.151447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:08.151455 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:08.151474 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:08.152986 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:08.153490 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:08.153496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:08.153497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:08.153579 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:08.153854 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:08.153859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:08.153861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:08.154065 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:08.154264 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:08.154432 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:18:08.154571 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":134,"threads_started":1}
W20260620 20:18:08.155468 3148 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57960
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[2998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[2998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:08.176736 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:08.169923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.2.235.193:44711)
0620 20:18:08.170063 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:08.170065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:08.170126 (+ 61us) server_negotiation.cc:407] Connection header received
0620 20:18:08.170208 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:08.170210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:08.170241 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:08.170298 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:08.170890 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.171371 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.172070 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:08.172258 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:08.172320 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:08.174313 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:08.174315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:08.174317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:08.174336 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:08.175608 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:08.175905 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:08.175907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:08.175908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:08.175966 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:08.176137 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:08.176138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:08.176139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:08.176269 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:08.176356 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:08.176498 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 20:18:08.176579 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:18:08.177562 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3105
I20260620 20:18:08.182765 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3239
I20260620 20:18:08.187832 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3373
I20260620 20:18:08.192456 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3011
2026-06-20T20:18:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1941 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[3516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[3516](info): set up 2 sockets
Jun 20 20:18:08 dist-test-slave-8z53 krb5kdc[3516](info): commencing operation
krb5kdc: starting...
W20260620 20:18:09.160811 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:09.262032 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34145
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:09.359249 3529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:09.359481 3529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:09.359545 3529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:09.362993 3529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:09.363061 3529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:09.363096 3529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:09.363153 3529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:09.363188 3529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:09.367756 3529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:09.368649 3529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:09.369452 3529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:09.376747 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:09.376740 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:09.376741 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:09.376808 3529 server_base.cc:1034] running on GCE node
I20260620 20:18:09.377493 3529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:09.378332 3529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:09.379484 3529 hybrid_clock.cc:648] HybridClock initialized: now 1781986689379464 us; error 34 us; skew 500 ppm
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:09.386802 3529 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:09.387528 3529 webserver.cc:460] Webserver started at http://127.2.235.254:40095/ using document root <none> and password file <none>
I20260620 20:18:09.387993 3529 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:09.388072 3529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:09.388273 3529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:09.390024 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "e7ab3053b607411184af1804ad33f5db"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "aaec7e888d1a3640a4ee15471d039fcc"
server_key_iv: "019aaebdff66f99a91ca2531a248a8e4"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.390555 3529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "e7ab3053b607411184af1804ad33f5db"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "aaec7e888d1a3640a4ee15471d039fcc"
server_key_iv: "019aaebdff66f99a91ca2531a248a8e4"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.393648 3529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:09.395680 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.396687 3529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:09.396804 3529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "e7ab3053b607411184af1804ad33f5db"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "aaec7e888d1a3640a4ee15471d039fcc"
server_key_iv: "019aaebdff66f99a91ca2531a248a8e4"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.396910 3529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:09.408133 3529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:09.410441 3529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:09.410607 3529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:09.416886 3529 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:34145
I20260620 20:18:09.416913 3596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:34145 every 8 connection(s)
I20260620 20:18:09.417644 3529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:09.419998 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:09.425057 3597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db: Bootstrap starting.
I20260620 20:18:09.426599 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3529
I20260620 20:18:09.426687 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:09.426913 2991 external_mini_cluster.cc:1428] Setting key 80c654a2a7301c6a8ec43f6d3729b5e6
I20260620 20:18:09.427325 3597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:09.428200 3597 log.cc:824] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:09.430015 3597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db: No bootstrap required, opened a new log
I20260620 20:18:09.433172 3597 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ab3053b607411184af1804ad33f5db" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34145 } }
I20260620 20:18:09.433452 3597 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:09.433517 3597 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ab3053b607411184af1804ad33f5db, State: Initialized, Role: FOLLOWER
I20260620 20:18:09.433923 3597 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ab3053b607411184af1804ad33f5db" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34145 } }
I20260620 20:18:09.434068 3597 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:09.434159 3597 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:09.434257 3597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 0 FOLLOWER]: Advancing to term 1
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:09.435351 3597 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ab3053b607411184af1804ad33f5db" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34145 } }
I20260620 20:18:09.435655 3597 leader_election.cc:302] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ab3053b607411184af1804ad33f5db; no voters:
I20260620 20:18:09.435954 3597 leader_election.cc:288] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:09.436015 3602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:09.436483 3602 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [term 1 LEADER]: Becoming Leader. State: Replica: e7ab3053b607411184af1804ad33f5db, State: Running, Role: LEADER
I20260620 20:18:09.437083 3602 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ab3053b607411184af1804ad33f5db" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34145 } }
I20260620 20:18:09.437160 3597 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:09.438563 3604 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ab3053b607411184af1804ad33f5db. Latest consensus state: current_term: 1 leader_uuid: "e7ab3053b607411184af1804ad33f5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ab3053b607411184af1804ad33f5db" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34145 } } }
I20260620 20:18:09.438687 3604 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:09.438974 3603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ab3053b607411184af1804ad33f5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ab3053b607411184af1804ad33f5db" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34145 } } }
I20260620 20:18:09.439064 3603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:09.441337 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:09.427860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.2.235.254:34145)
0620 20:18:09.428253 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:09.428266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:09.428295 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:18:09.429009 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:09.429026 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:09.429346 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:09.429631 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:09.430316 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.431162 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.431980 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.432420 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.432572 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:09.435790 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:09.435797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:09.435811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:09.435843 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:09.437802 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:09.438297 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:09.438305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:09.438313 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:09.438446 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:09.438811 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:09.438815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:09.438817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:09.439274 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:09.439444 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:09.440273 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 20:18:09.440468 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":128,"threads_started":1}
W20260620 20:18:09.441845 3617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:09.441919 3617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:09.442003 3616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:09.442605 3616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:09.447077 3616 catalog_manager.cc:1347] Generated new cluster ID: 448518794e154da5adc2ffc3e98f0168
I20260620 20:18:09.447135 3616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:09.459179 3616 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:09.459896 3616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:09.465446 3616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ab3053b607411184af1804ad33f5db: Generated new TSK 0
I20260620 20:18:09.465953 3616 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:09.498121 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34145
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:09.616544 3623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:09.616760 3623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:09.616827 3623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:09.620529 3623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:09.620579 3623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:09.620675 3623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:09.625516 3623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:09.626428 3623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:09.627364 3623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:09.636042 3628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:09.636107 3629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:09.636049 3631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:09.636389 3623 server_base.cc:1034] running on GCE node
I20260620 20:18:09.636768 3623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:09.637400 3623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:09.638566 3623 hybrid_clock.cc:648] HybridClock initialized: now 1781986689638549 us; error 52 us; skew 500 ppm
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:09.641042 3623 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:09.641922 3623 webserver.cc:460] Webserver started at http://127.2.235.193:39799/ using document root <none> and password file <none>
I20260620 20:18:09.642427 3623 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:09.642514 3623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:09.642710 3623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:09.644493 3623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "50d451470b7f42bcafdc7e7bb4b23046"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "e5397de8ecd44a9ccba782a33c88c1ab"
server_key_iv: "d957cfcb8296d51ef992e2f2480e619f"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.645064 3623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "50d451470b7f42bcafdc7e7bb4b23046"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "e5397de8ecd44a9ccba782a33c88c1ab"
server_key_iv: "d957cfcb8296d51ef992e2f2480e619f"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.648233 3623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:09.650337 3638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.651355 3623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:09.651472 3623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "50d451470b7f42bcafdc7e7bb4b23046"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "e5397de8ecd44a9ccba782a33c88c1ab"
server_key_iv: "d957cfcb8296d51ef992e2f2480e619f"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.651566 3623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:09.663995 3623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:09.666901 3623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:09.667081 3623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:09.667562 3623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:09.668251 3623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:09.668318 3623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.668380 3623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:09.668426 3623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.677065 3623 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:32861
I20260620 20:18:09.677279 3751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:32861 every 8 connection(s)
I20260620 20:18:09.677935 3623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:09.683833 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3623
I20260620 20:18:09.683919 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:09.684175 2991 external_mini_cluster.cc:1428] Setting key cf1357c2c6fe60b6e18da88916a2eb81
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:09.692047 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:09.679548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:47999 (local address 127.2.235.254:34145)
0620 20:18:09.679678 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:09.679684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:09.680621 (+ 937us) server_negotiation.cc:407] Connection header received
0620 20:18:09.682375 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:09.682379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:09.682461 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:09.682548 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:09.683631 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.684326 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.685714 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.685924 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.685995 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:09.688497 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:09.688500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:09.688502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:09.688521 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:09.690139 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:09.690609 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:09.690623 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:09.690626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:09.690732 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:09.691029 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:09.691033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:09.691037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:09.691277 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:09.691374 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:09.691759 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 20:18:09.691878 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:18:09.692477 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:09.679956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34145 (local address 127.2.235.193:47999)
0620 20:18:09.680459 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:09.680506 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:09.682083 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:09.682743 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:09.682752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:09.683075 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:09.683480 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:09.683492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.684442 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.684446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:09.685595 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:09.685605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.686060 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.686067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:09.686281 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:09.686788 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:09.686829 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:09.688353 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:09.690266 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:09.690270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:09.690279 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:09.690511 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:09.690824 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:09.690826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:09.690827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:09.690943 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:09.691371 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:09.691375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:09.691550 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 20:18:09.691686 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:18:09.691880 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:09.693548 3752 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34145
I20260620 20:18:09.693797 3752 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:09.694238 3752 heartbeater.cc:511] Master 127.2.235.254:34145 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:18:09.695776 3561 ts_manager.cc:194] Registered new tserver with Master: 50d451470b7f42bcafdc7e7bb4b23046 (127.2.235.193:32861)
I20260620 20:18:09.697338 3561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:47999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:09.709035 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34145
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:09.806227 3757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:09.806450 3757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:09.806517 3757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:09.810041 3757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:09.810127 3757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:09.810206 3757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:09.814885 3757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:09.815653 3757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:09.816464 3757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:09.824985 3762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:09.825034 3763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:09.825049 3757 server_base.cc:1034] running on GCE node
W20260620 20:18:09.824988 3765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:09.825609 3757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:09.826151 3757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:09.827350 3757 hybrid_clock.cc:648] HybridClock initialized: now 1781986689827322 us; error 41 us; skew 500 ppm
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:09.829875 3757 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:09.830576 3757 webserver.cc:460] Webserver started at http://127.2.235.194:44703/ using document root <none> and password file <none>
I20260620 20:18:09.831035 3757 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:09.831121 3757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:09.831312 3757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:09.832967 3757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "4b87422367df4efdb017c9505fa9bf93"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "4359adf4983afa57c2105140a6a34ada"
server_key_iv: "ecf2c8a92247e38ea4946575a882cd4b"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.833496 3757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "4b87422367df4efdb017c9505fa9bf93"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "4359adf4983afa57c2105140a6a34ada"
server_key_iv: "ecf2c8a92247e38ea4946575a882cd4b"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.836680 3757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:18:09.839000 3772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.840091 3757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:09.840229 3757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "4b87422367df4efdb017c9505fa9bf93"
format_stamp: "Formatted at 2026-06-20 20:18:09 on dist-test-slave-8z53"
server_key: "4359adf4983afa57c2105140a6a34ada"
server_key_iv: "ecf2c8a92247e38ea4946575a882cd4b"
server_key_version: "encryptionkey@0"
I20260620 20:18:09.840339 3757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:09.859447 3757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:09.862282 3757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:09.862452 3757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:09.862922 3757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:09.863659 3757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:09.863726 3757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.863790 3757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:09.863842 3757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:09.871408 3757 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:46507
I20260620 20:18:09.871434 3885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:46507 every 8 connection(s)
I20260620 20:18:09.872135 3757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:09.873442 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3757
I20260620 20:18:09.873562 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:09.873876 2991 external_mini_cluster.cc:1428] Setting key 697387deb210d07de83a7b6a8c8960f0
Jun 20 20:18:09 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:18:09.885586 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:09.873737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:48711 (local address 127.2.235.254:34145)
0620 20:18:09.873888 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:09.873893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:09.874706 (+ 813us) server_negotiation.cc:407] Connection header received
0620 20:18:09.876041 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:09.876047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:09.876105 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:09.876180 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:09.877386 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.877853 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.879459 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.879647 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.879731 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:09.882243 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:09.882246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:09.882248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:09.882269 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:09.883628 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:09.884139 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:09.884144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:09.884146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:09.884231 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:09.884556 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:09.884560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:09.884562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:09.884731 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:09.884831 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:09.885302 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 20:18:09.885411 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:18:09.886289 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:09.874132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34145 (local address 127.2.235.194:48711)
0620 20:18:09.874541 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:09.874602 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:09.875849 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:09.876290 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:09.876296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:09.876568 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:09.877213 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:09.877232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.877969 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.877975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:09.879338 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:09.879351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:09.879785 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:09.879792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:09.879998 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:09.880514 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:09.880537 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:09.882058 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:09.883778 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:09.883784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:09.883796 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:09.884036 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:09.884328 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:09.884330 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:09.884333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:09.884461 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:09.884839 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:09.884848 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:09.885056 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 20:18:09.885229 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 20:18:09.885536 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260620 20:18:09.887271 3886 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34145
I20260620 20:18:09.887483 3886 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:09.887969 3886 heartbeater.cc:511] Master 127.2.235.254:34145 requested a full tablet report, sending...
I20260620 20:18:09.888937 3561 ts_manager.cc:194] Registered new tserver with Master: 4b87422367df4efdb017c9505fa9bf93 (127.2.235.194:46507)
I20260620 20:18:09.889778 3561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:48711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:09.898000 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34145
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:09.999181 3891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:09.999399 3891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:09.999475 3891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:10.003149 3891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:10.003216 3891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:10.003298 3891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:10.008087 3891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34145
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:10.008888 3891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:10.009757 3891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:10.017988 3899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:10.017972 3896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:10.017998 3897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:10.018201 3891 server_base.cc:1034] running on GCE node
I20260620 20:18:10.018579 3891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:10.019140 3891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:10.020360 3891 hybrid_clock.cc:648] HybridClock initialized: now 1781986690020342 us; error 42 us; skew 500 ppm
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:10.022958 3891 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:10.023701 3891 webserver.cc:460] Webserver started at http://127.2.235.195:33261/ using document root <none> and password file <none>
I20260620 20:18:10.024195 3891 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:10.024281 3891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:10.024490 3891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:10.026322 3891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "592650909c3242fb815fdd7a28423a75"
format_stamp: "Formatted at 2026-06-20 20:18:10 on dist-test-slave-8z53"
server_key: "9ac031282afa660934a63d14722a6a04"
server_key_iv: "d0cea4c729513f6502a76fba22505d05"
server_key_version: "encryptionkey@0"
I20260620 20:18:10.026836 3891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "592650909c3242fb815fdd7a28423a75"
format_stamp: "Formatted at 2026-06-20 20:18:10 on dist-test-slave-8z53"
server_key: "9ac031282afa660934a63d14722a6a04"
server_key_iv: "d0cea4c729513f6502a76fba22505d05"
server_key_version: "encryptionkey@0"
I20260620 20:18:10.030174 3891 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:10.032307 3906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:10.033677 3891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:10.033815 3891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "592650909c3242fb815fdd7a28423a75"
format_stamp: "Formatted at 2026-06-20 20:18:10 on dist-test-slave-8z53"
server_key: "9ac031282afa660934a63d14722a6a04"
server_key_iv: "d0cea4c729513f6502a76fba22505d05"
server_key_version: "encryptionkey@0"
I20260620 20:18:10.033917 3891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:10.056260 3891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:10.058830 3891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:10.059005 3891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:10.059476 3891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:10.060158 3891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:10.060240 3891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:10.060302 3891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:10.060348 3891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:10.067982 3891 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:45713
I20260620 20:18:10.068005 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:45713 every 8 connection(s)
I20260620 20:18:10.068638 3891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:10.072672 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 3891
I20260620 20:18:10.072767 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:10.073021 2991 external_mini_cluster.cc:1428] Setting key b0ea1b0200d04c231e8c173e5800402e
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986690, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:10.080431 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.070210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:57967 (local address 127.2.235.254:34145)
0620 20:18:10.070327 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.070330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.070934 (+ 604us) server_negotiation.cc:407] Connection header received
0620 20:18:10.072098 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.072102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.072170 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.072260 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:10.073420 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.073959 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.075058 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.075278 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.075376 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.077404 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:10.077423 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:10.077425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:10.077444 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:10.078603 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.079070 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.079074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.079076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.079159 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.079472 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.079476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.079478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.079636 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:10.079732 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.080157 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.080273 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:10.080996 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.070380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34145 (local address 127.2.235.195:57967)
0620 20:18:10.070787 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:10.070820 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:10.071912 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:10.072401 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:10.072407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:10.072768 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:10.073283 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.073303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.074076 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.074080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.074933 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.074942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.075404 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.075408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.075561 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.076050 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:10.076064 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:10.077262 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:10.078711 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:10.078716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:10.078728 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:10.078960 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:10.079250 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:10.079255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:10.079257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:10.079376 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:10.079741 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:10.079749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:10.079946 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 20:18:10.080100 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:18:10.080377 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260620 20:18:10.081938 4020 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34145
I20260620 20:18:10.082152 4020 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:10.082604 4020 heartbeater.cc:511] Master 127.2.235.254:34145 requested a full tablet report, sending...
I20260620 20:18:10.083487 3561 ts_manager.cc:194] Registered new tserver with Master: 592650909c3242fb815fdd7a28423a75 (127.2.235.195:45713)
I20260620 20:18:10.084017 3561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:57967
I20260620 20:18:10.086344 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:10.095326 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.089200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.2.235.254:34145)
0620 20:18:10.089331 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.089334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.089534 (+ 200us) server_negotiation.cc:407] Connection header received
0620 20:18:10.089618 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.089620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.089650 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.089714 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:10.090268 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.090699 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.091448 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.091637 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.091707 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.092817 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:10.092819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:10.092821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:10.092840 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:10.094065 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.094412 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.094414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.094415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.094479 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.094703 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.094705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.094706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.094895 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:10.094986 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.095144 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.095226 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:18:10.098991 3561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:10.101881 3561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:10.114063 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.108919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:46507 (local address 127.0.0.1:46700)
0620 20:18:10.109559 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:10.109572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:10.109680 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:10.110470 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:10.110473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:10.110489 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:10.110586 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.110592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.111673 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.111676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.112594 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.112603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.113545 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.113548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.113660 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.113693 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:18:10.113828 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 20:18:10.113933 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260620 20:18:10.114095 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.109544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.2.235.194:46507)
0620 20:18:10.109924 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.109934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.110075 (+ 141us) server_negotiation.cc:407] Connection header received
0620 20:18:10.110152 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.110157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.110333 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.110719 (+ 386us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:10.110878 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.111522 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.112728 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.113435 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.113545 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.113699 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.113859 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.113953 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":108,"threads_started":1}
I20260620 20:18:10.114460 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.108814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:45713 (local address 127.0.0.1:35548)
0620 20:18:10.109523 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:10.109563 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:10.109682 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:10.110320 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:10.110325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:10.110344 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:10.110453 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.110459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.111595 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.111601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.112662 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.112671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.113752 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.113756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.113842 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.113863 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:18:10.113936 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:18:10.114004 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":42,"thread_start_us":81,"threads_started":1}
I20260620 20:18:10.114472 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.109196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:32861 (local address 127.0.0.1:41584)
0620 20:18:10.109947 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:10.109961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:10.110034 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:10.110764 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:10.110767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:10.110785 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:10.110846 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.110852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.112086 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.112089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.113021 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.113028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.114127 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.114130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.114224 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.114244 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:18:10.114310 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:18:10.114369 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":85,"threads_started":1}
I20260620 20:18:10.114804 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.109558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.2.235.193:32861)
0620 20:18:10.110238 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.110248 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.110265 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:18:10.110324 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.110329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.110470 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.110582 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:10.110969 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.111955 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.113128 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.113867 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.113958 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.114070 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.114534 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.114640 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":112,"threads_started":1}
I20260620 20:18:10.114991 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.108882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.2.235.195:45713)
0620 20:18:10.109483 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.109493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.109796 (+ 303us) server_negotiation.cc:407] Connection header received
0620 20:18:10.109854 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.109862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.110021 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.110178 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:10.110562 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.111465 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.112787 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.113466 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.113549 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.113689 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.114131 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.114216 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":127,"threads_started":1}
I20260620 20:18:10.116649 3820 tablet_service.cc:1467] Processing CreateTablet for tablet b58e181ca16549aea5080fc6f3b47e0b (DEFAULT_TABLE table=test-table [id=f201dbe970c7436aa085cac5c99c582c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:10.116976 3954 tablet_service.cc:1467] Processing CreateTablet for tablet b58e181ca16549aea5080fc6f3b47e0b (DEFAULT_TABLE table=test-table [id=f201dbe970c7436aa085cac5c99c582c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:10.116977 3686 tablet_service.cc:1467] Processing CreateTablet for tablet b58e181ca16549aea5080fc6f3b47e0b (DEFAULT_TABLE table=test-table [id=f201dbe970c7436aa085cac5c99c582c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:10.117599 3820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58e181ca16549aea5080fc6f3b47e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:10.117897 3686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58e181ca16549aea5080fc6f3b47e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:10.117897 3954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58e181ca16549aea5080fc6f3b47e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:10.122838 4037 tablet_bootstrap.cc:492] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93: Bootstrap starting.
I20260620 20:18:10.123873 4038 tablet_bootstrap.cc:492] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75: Bootstrap starting.
I20260620 20:18:10.124272 4039 tablet_bootstrap.cc:492] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: Bootstrap starting.
I20260620 20:18:10.126518 4037 tablet_bootstrap.cc:654] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:10.126634 4039 tablet_bootstrap.cc:654] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:10.126952 4038 tablet_bootstrap.cc:654] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:10.127390 4037 log.cc:824] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:10.127390 4039 log.cc:824] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:10.127596 4038 log.cc:824] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:10.128914 4039 tablet_bootstrap.cc:492] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: No bootstrap required, opened a new log
I20260620 20:18:10.129084 4039 ts_tablet_manager.cc:1397] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:18:10.129094 4037 tablet_bootstrap.cc:492] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93: No bootstrap required, opened a new log
I20260620 20:18:10.129094 4038 tablet_bootstrap.cc:492] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75: No bootstrap required, opened a new log
I20260620 20:18:10.129243 4037 ts_tablet_manager.cc:1397] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 20:18:10.129266 4038 ts_tablet_manager.cc:1397] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 20:18:10.131850 4039 raft_consensus.cc:348] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.132076 4039 raft_consensus.cc:374] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:10.132143 4039 raft_consensus.cc:729] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d451470b7f42bcafdc7e7bb4b23046, State: Initialized, Role: FOLLOWER
I20260620 20:18:10.132552 4039 consensus_queue.cc:260] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.132767 4038 raft_consensus.cc:348] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.132767 4037 raft_consensus.cc:348] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.133000 4037 raft_consensus.cc:374] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:10.133059 4038 raft_consensus.cc:374] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:10.133095 4037 raft_consensus.cc:729] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b87422367df4efdb017c9505fa9bf93, State: Initialized, Role: FOLLOWER
I20260620 20:18:10.133104 3752 heartbeater.cc:503] Master 127.2.235.254:34145 was elected leader, sending a full tablet report...
I20260620 20:18:10.133165 4038 raft_consensus.cc:729] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592650909c3242fb815fdd7a28423a75, State: Initialized, Role: FOLLOWER
I20260620 20:18:10.133510 4039 ts_tablet_manager.cc:1428] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260620 20:18:10.133657 4038 consensus_queue.cc:260] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.133644 4037 consensus_queue.cc:260] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.134363 4020 heartbeater.cc:503] Master 127.2.235.254:34145 was elected leader, sending a full tablet report...
I20260620 20:18:10.134472 3886 heartbeater.cc:503] Master 127.2.235.254:34145 was elected leader, sending a full tablet report...
I20260620 20:18:10.134603 4038 ts_tablet_manager.cc:1428] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:18:10.134868 4037 ts_tablet_manager.cc:1428] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260620 20:18:10.178943 3753 tablet.cc:2369] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:10.181963 4044 raft_consensus.cc:482] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:10.182104 4044 raft_consensus.cc:504] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.183075 4044 leader_election.cc:288] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b87422367df4efdb017c9505fa9bf93 (127.2.235.194:46507), 50d451470b7f42bcafdc7e7bb4b23046 (127.2.235.193:32861)
I20260620 20:18:10.186879 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.183275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:32861 (local address 127.2.235.195:55457)
0620 20:18:10.183366 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:10.183382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:10.183455 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:10.183747 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:10.183750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:10.183766 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:10.183880 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.183887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.184789 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.184792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.185827 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.185837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.186471 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.186474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.186592 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.186609 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:18:10.186722 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:18:10.186789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260620 20:18:10.186939 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.183374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:55457 (local address 127.2.235.193:32861)
0620 20:18:10.183494 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.183498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.183513 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:18:10.183556 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.183559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.183604 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.183727 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:10.184005 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.184688 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.185936 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.186364 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.186467 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.186508 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.186753 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.186819 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 20:18:10.187458 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.183275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:46507 (local address 127.2.235.195:42875)
0620 20:18:10.183918 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:10.183929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:10.184040 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:10.184327 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:10.184331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:10.184347 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:10.184462 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.184467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.185202 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.185205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.186139 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:10.186145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.187030 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.187037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:10.187145 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.187162 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:18:10.187251 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:18:10.187312 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":249,"thread_start_us":147,"threads_started":1}
I20260620 20:18:10.187568 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.183447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:42875 (local address 127.2.235.194:46507)
0620 20:18:10.183562 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.183567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.184008 (+ 441us) server_negotiation.cc:407] Connection header received
0620 20:18:10.184152 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.184155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.184199 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.184267 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:10.184559 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.185092 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.186233 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.186811 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.186913 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.186957 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.187388 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.187460 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:18:10.187600 3706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58e181ca16549aea5080fc6f3b47e0b" candidate_uuid: "592650909c3242fb815fdd7a28423a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d451470b7f42bcafdc7e7bb4b23046" is_pre_election: true
I20260620 20:18:10.187882 3706 raft_consensus.cc:2454] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592650909c3242fb815fdd7a28423a75 in term 0.
I20260620 20:18:10.188251 3909 leader_election.cc:302] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d451470b7f42bcafdc7e7bb4b23046, 592650909c3242fb815fdd7a28423a75; no voters:
I20260620 20:18:10.188508 4044 raft_consensus.cc:2781] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:10.188529 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58e181ca16549aea5080fc6f3b47e0b" candidate_uuid: "592650909c3242fb815fdd7a28423a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87422367df4efdb017c9505fa9bf93" is_pre_election: true
I20260620 20:18:10.188598 4044 raft_consensus.cc:482] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:10.188643 4044 raft_consensus.cc:3037] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:10.188767 3840 raft_consensus.cc:2454] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592650909c3242fb815fdd7a28423a75 in term 0.
I20260620 20:18:10.189723 4044 raft_consensus.cc:504] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.189967 4044 leader_election.cc:288] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [CANDIDATE]: Term 1 election: Requested vote from peers 4b87422367df4efdb017c9505fa9bf93 (127.2.235.194:46507), 50d451470b7f42bcafdc7e7bb4b23046 (127.2.235.193:32861)
I20260620 20:18:10.190281 3706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58e181ca16549aea5080fc6f3b47e0b" candidate_uuid: "592650909c3242fb815fdd7a28423a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50d451470b7f42bcafdc7e7bb4b23046"
I20260620 20:18:10.190281 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58e181ca16549aea5080fc6f3b47e0b" candidate_uuid: "592650909c3242fb815fdd7a28423a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87422367df4efdb017c9505fa9bf93"
I20260620 20:18:10.190398 3840 raft_consensus.cc:3037] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:10.190418 3706 raft_consensus.cc:3037] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:10.191592 3706 raft_consensus.cc:2454] T b58e181ca16549aea5080fc6f3b47e0b P 50d451470b7f42bcafdc7e7bb4b23046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592650909c3242fb815fdd7a28423a75 in term 1.
I20260620 20:18:10.191592 3840 raft_consensus.cc:2454] T b58e181ca16549aea5080fc6f3b47e0b P 4b87422367df4efdb017c9505fa9bf93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592650909c3242fb815fdd7a28423a75 in term 1.
I20260620 20:18:10.191920 3907 leader_election.cc:302] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b87422367df4efdb017c9505fa9bf93, 592650909c3242fb815fdd7a28423a75; no voters:
I20260620 20:18:10.192092 4044 raft_consensus.cc:2781] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:10.192299 4044 raft_consensus.cc:686] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [term 1 LEADER]: Becoming Leader. State: Replica: 592650909c3242fb815fdd7a28423a75, State: Running, Role: LEADER
I20260620 20:18:10.192564 4044 consensus_queue.cc:237] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } }
I20260620 20:18:10.195149 3560 catalog_manager.cc:5495] T b58e181ca16549aea5080fc6f3b47e0b P 592650909c3242fb815fdd7a28423a75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 592650909c3242fb815fdd7a28423a75 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "592650909c3242fb815fdd7a28423a75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "592650909c3242fb815fdd7a28423a75" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b87422367df4efdb017c9505fa9bf93" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d451470b7f42bcafdc7e7bb4b23046" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 32861 } health_report { overall_health: UNKNOWN } } }
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:10.212147 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.204286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.2.235.193:32861)
0620 20:18:10.204401 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.204405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.204568 (+ 163us) server_negotiation.cc:407] Connection header received
0620 20:18:10.204723 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.204727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.204769 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.204850 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:10.205483 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.205872 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.206626 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.206836 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.206938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.209189 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:10.209197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:10.209206 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:10.209226 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:10.210691 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.211048 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.211052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.211053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.211154 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.211364 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.211366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.211367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.211563 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:10.211709 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.211852 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.211977 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986690, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986690, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:10.234705 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.228035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.2.235.193:32861)
0620 20:18:10.228266 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.228268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.228278 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:18:10.228303 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.228304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.228329 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.228383 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:10.228919 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.229367 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.229997 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.230173 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.230237 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.232110 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:10.232112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:10.232114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:10.232132 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:10.233388 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.233729 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.233731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.233732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.233797 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.233999 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.234001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.234002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.234128 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:10.234233 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.234428 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.234508 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[3516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:10.257078 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:10.249383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.2.235.193:32861)
0620 20:18:10.249523 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:10.249528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:10.249677 (+ 149us) server_negotiation.cc:407] Connection header received
0620 20:18:10.249828 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:10.249832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:10.249877 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:10.249965 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:10.250641 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.251061 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.251843 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:10.252057 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:10.252175 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:10.254312 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:10.254316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:10.254317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:10.254337 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:10.255524 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.255912 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.255917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.255918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.255990 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:10.256297 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:10.256300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:10.256302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:10.256441 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:10.256586 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:10.256814 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:18:10.256929 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 20:18:10.258210 3666 tablet_service.cc:2908] Rejecting scan request for tablet b58e181ca16549aea5080fc6f3b47e0b: Uninitialized: clean time has not yet been initialized
I20260620 20:18:10.259905 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3623
I20260620 20:18:10.265275 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3757
I20260620 20:18:10.270046 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3891
I20260620 20:18:10.275005 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 3529
2026-06-20T20:18:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2082 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[4061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[4061](info): set up 2 sockets
Jun 20 20:18:10 dist-test-slave-8z53 krb5kdc[4061](info): commencing operation
krb5kdc: starting...
W20260620 20:18:11.240782 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:11.343623 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:35977
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:35977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:11.440716 4074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:11.440909 4074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:11.440944 4074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:11.444543 4074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:11.444583 4074 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:18:11.444600 4074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:11.444623 4074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:11.444640 4074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:11.444655 4074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:11.449189 4074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:35977
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:11.450033 4074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:11.450894 4074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:11.458463 4082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:11.458490 4080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:11.458498 4079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:11.458642 4074 server_base.cc:1034] running on GCE node
I20260620 20:18:11.459276 4074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:11.460244 4074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:11.461397 4074 hybrid_clock.cc:648] HybridClock initialized: now 1781986691461385 us; error 22 us; skew 500 ppm
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:11.463701 4074 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:11.464481 4074 webserver.cc:460] Webserver started at http://127.2.235.254:33057/ using document root <none> and password file <none>
I20260620 20:18:11.464996 4074 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:11.465070 4074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:11.465302 4074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:11.467020 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "7f43ac54dc544e41ac2d473af15eb09a"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "91e8245fbffbd71547cf2081a29b3bd5"
server_key_iv: "b1f92ba7d654d05984565c3c3ab44738"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.467530 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "7f43ac54dc544e41ac2d473af15eb09a"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "91e8245fbffbd71547cf2081a29b3bd5"
server_key_iv: "b1f92ba7d654d05984565c3c3ab44738"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.470724 4074 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:18:11.472829 4089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:11.473990 4074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:18:11.474143 4074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "7f43ac54dc544e41ac2d473af15eb09a"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "91e8245fbffbd71547cf2081a29b3bd5"
server_key_iv: "b1f92ba7d654d05984565c3c3ab44738"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.474246 4074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:11.498693 4074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:11.501291 4074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:11.501466 4074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:11.508646 4074 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:35977
I20260620 20:18:11.508675 4151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:35977 every 8 connection(s)
I20260620 20:18:11.509488 4074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:11.511855 4152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:11.516733 4152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: Bootstrap starting.
I20260620 20:18:11.518261 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4074
I20260620 20:18:11.518383 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:11.518630 2991 external_mini_cluster.cc:1428] Setting key bbc20e7595d1fd3f6de50aab88b111ff
I20260620 20:18:11.519178 4152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:11.520032 4152 log.cc:824] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:11.521836 4152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: No bootstrap required, opened a new log
I20260620 20:18:11.524967 4152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43ac54dc544e41ac2d473af15eb09a" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35977 } }
I20260620 20:18:11.525228 4152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:11.525344 4152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f43ac54dc544e41ac2d473af15eb09a, State: Initialized, Role: FOLLOWER
I20260620 20:18:11.525745 4152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43ac54dc544e41ac2d473af15eb09a" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35977 } }
I20260620 20:18:11.525859 4152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:11.525944 4152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:11.526054 4152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 0 FOLLOWER]: Advancing to term 1
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:11.527058 4152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43ac54dc544e41ac2d473af15eb09a" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35977 } }
I20260620 20:18:11.527446 4152 leader_election.cc:302] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f43ac54dc544e41ac2d473af15eb09a; no voters:
I20260620 20:18:11.527737 4152 leader_election.cc:288] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:11.527848 4157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:11.528074 4157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [term 1 LEADER]: Becoming Leader. State: Replica: 7f43ac54dc544e41ac2d473af15eb09a, State: Running, Role: LEADER
I20260620 20:18:11.528376 4157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43ac54dc544e41ac2d473af15eb09a" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35977 } }
I20260620 20:18:11.528908 4152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:11.529598 4157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f43ac54dc544e41ac2d473af15eb09a. Latest consensus state: current_term: 1 leader_uuid: "7f43ac54dc544e41ac2d473af15eb09a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43ac54dc544e41ac2d473af15eb09a" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35977 } } }
I20260620 20:18:11.529718 4157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:11.530090 4165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:11.530567 4158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f43ac54dc544e41ac2d473af15eb09a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43ac54dc544e41ac2d473af15eb09a" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35977 } } }
I20260620 20:18:11.530678 4158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:11.533155 4165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:11.533340 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.519629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.2.235.254:35977)
0620 20:18:11.519992 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:11.520005 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:11.520036 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:18:11.520861 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:11.520878 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:11.521162 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:11.521509 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:11.522298 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.523194 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.524026 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.524480 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.524639 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.527471 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:11.527476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:11.527487 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:11.527510 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:11.529626 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.530793 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.530799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.530804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.530934 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.531268 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.531272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.531277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.531720 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:11.531919 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:11.532206 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 20:18:11.532468 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260620 20:18:11.538203 4165 catalog_manager.cc:1347] Generated new cluster ID: 17893a9a4b1b4343a7a1f1191b784004
I20260620 20:18:11.538259 4165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:11.563761 4165 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:11.564569 4165 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:11.570024 4165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: Generated new TSK 0
I20260620 20:18:11.570567 4165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:11.598206 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:35977
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:11.700810 4178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:11.701009 4178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:11.701045 4178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:11.705065 4178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:11.705109 4178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:11.705174 4178 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:11.705248 4178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:11.709822 4178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:35977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:11.710632 4178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:11.711494 4178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:11.719875 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:11.719882 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:11.719895 4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:11.720031 4178 server_base.cc:1034] running on GCE node
I20260620 20:18:11.720428 4178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:11.720993 4178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:11.722194 4178 hybrid_clock.cc:648] HybridClock initialized: now 1781986691722161 us; error 45 us; skew 500 ppm
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:11.724709 4178 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:11.725538 4178 webserver.cc:460] Webserver started at http://127.2.235.193:38209/ using document root <none> and password file <none>
I20260620 20:18:11.726015 4178 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:11.726096 4178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:11.726290 4178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:11.727974 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "856582ff2db7443daa5ac02c67a6b29a"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "83a676ba58b3d9a61e85df016a5af0f8"
server_key_iv: "f0a290ddf30b376dd34535c6295af415"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.728521 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "856582ff2db7443daa5ac02c67a6b29a"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "83a676ba58b3d9a61e85df016a5af0f8"
server_key_iv: "f0a290ddf30b376dd34535c6295af415"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.731966 4178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:18:11.734242 4193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:11.735329 4178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:11.735469 4178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "856582ff2db7443daa5ac02c67a6b29a"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "83a676ba58b3d9a61e85df016a5af0f8"
server_key_iv: "f0a290ddf30b376dd34535c6295af415"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.735569 4178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:11.754834 4178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:11.757248 4178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:11.757475 4178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:11.758649 4178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:11.758716 4178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:11.758774 4178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:11.758821 4178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:11.768268 4178 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40733
I20260620 20:18:11.768443 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40733 every 8 connection(s)
I20260620 20:18:11.769121 4178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:11.771631 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.759377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:40027 (local address 127.2.235.254:35977)
0620 20:18:11.759540 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:11.759545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:11.760563 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 20:18:11.761575 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:11.761578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:11.761634 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:11.761717 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:11.763052 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.763958 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.765703 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.765908 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.765985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.768650 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:11.768653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:11.768658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:11.768684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:11.770108 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.770479 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.770482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.770483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.770569 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.770824 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.770827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.770828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.770987 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:11.771069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:11.771424 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 20:18:11.771515 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:18:11.772260 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.760034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.2.235.193:40027)
0620 20:18:11.760413 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:11.760454 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:11.761383 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:11.761864 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:11.761870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:11.762330 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:11.762910 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.762928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.764113 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.764116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.765586 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.765596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.766063 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.766068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.766268 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.766816 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:11.766836 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:11.768450 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:11.770193 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.770196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.770204 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.770402 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.770625 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.770626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.770627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.770724 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.771076 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:11.771080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:11.771247 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 20:18:11.771374 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 20:18:11.771561 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
I20260620 20:18:11.773290 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4178
I20260620 20:18:11.773411 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:11.773422 4309 heartbeater.cc:347] Connected to a master server at 127.2.235.254:35977
I20260620 20:18:11.773537 4309 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:11.773682 2991 external_mini_cluster.cc:1428] Setting key a98c5c907299f38c34aff52b4070dad2
I20260620 20:18:11.774034 4309 heartbeater.cc:511] Master 127.2.235.254:35977 requested a full tablet report, sending...
I20260620 20:18:11.775548 4106 ts_manager.cc:194] Registered new tserver with Master: 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193:40733)
I20260620 20:18:11.776808 4106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:40027
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:11.783550 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.775431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.2.235.193:56871)
0620 20:18:11.775709 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:11.775723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:11.775796 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:11.777065 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:11.777067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:11.777301 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:11.777396 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.777401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.778239 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.778244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.779194 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.779207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.779597 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.779604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.779715 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.780110 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:11.780127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:11.780543 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:11.782075 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.782078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.782079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.782317 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.782542 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.782544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.782546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.782636 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.783139 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:11.783145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:11.783255 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 20:18:11.783340 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:18:11.783428 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260620 20:18:11.783569 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.775503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:56871 (local address 127.2.235.254:35977)
0620 20:18:11.776769 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:11.776772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:11.776783 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:11.776808 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:11.776810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:11.776861 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:11.776962 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:11.777493 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.778105 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.779304 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.779483 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.779553 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.780686 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:11.780689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:11.780690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:11.780708 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:11.781973 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.782403 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.782405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.782405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.782461 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.782815 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.782818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.782819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.783044 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:11.783119 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:11.783394 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 20:18:11.783477 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:11.798106 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:35977
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:11.901288 4318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:11.901487 4318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:11.901526 4318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:11.905490 4318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:11.905534 4318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:11.905566 4318 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:11.905596 4318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:11.909912 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:35977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:11.910666 4318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:11.911463 4318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:11.919937 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:11.920023 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:11.920006 4318 server_base.cc:1034] running on GCE node
W20260620 20:18:11.920262 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:11.920598 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:11.921149 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:11.922308 4318 hybrid_clock.cc:648] HybridClock initialized: now 1781986691922291 us; error 26 us; skew 500 ppm
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:11.924782 4318 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:11.925753 4318 webserver.cc:460] Webserver started at http://127.2.235.194:37295/ using document root <none> and password file <none>
I20260620 20:18:11.926275 4318 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:11.926329 4318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:11.926515 4318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:11.928283 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "69b655e52ad942f6a244af35aac2bc26"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "25735abf866b596483a711ecc72ee44f"
server_key_iv: "71866ae2fe8290e33b002b1d422cbda5"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.928789 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "69b655e52ad942f6a244af35aac2bc26"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "25735abf866b596483a711ecc72ee44f"
server_key_iv: "71866ae2fe8290e33b002b1d422cbda5"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.932338 4318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:18:11.934589 4333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:11.935673 4318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:11.935788 4318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "69b655e52ad942f6a244af35aac2bc26"
format_stamp: "Formatted at 2026-06-20 20:18:11 on dist-test-slave-8z53"
server_key: "25735abf866b596483a711ecc72ee44f"
server_key_iv: "71866ae2fe8290e33b002b1d422cbda5"
server_key_version: "encryptionkey@0"
I20260620 20:18:11.935878 4318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:11.951862 4318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:11.954353 4318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:11.954528 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:11.955746 4318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:11.955790 4318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:11.955824 4318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:11.955884 4318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:11 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:11.966140 4318 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:40745
I20260620 20:18:11.966223 4448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:40745 every 8 connection(s)
I20260620 20:18:11.966926 4318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:11.969429 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.956553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:48307 (local address 127.2.235.254:35977)
0620 20:18:11.956680 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:11.956684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:11.957631 (+ 947us) server_negotiation.cc:407] Connection header received
0620 20:18:11.958615 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:11.958621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:11.958681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:11.958766 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:11.960328 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.961130 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.962963 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.963294 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.963375 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.966051 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:11.966057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:11.966059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:11.966123 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:11.967767 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.968156 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.968159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.968160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.968231 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.968495 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.968497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.968498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.968665 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:11.968737 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:11.969173 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 20:18:11.969302 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:18:11.970142 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.957029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.2.235.194:48307)
0620 20:18:11.957488 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:11.957524 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:11.958385 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:11.959199 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:11.959207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:11.959603 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:11.960207 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.960225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.961270 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.961276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.962827 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.962841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.963448 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.963454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.963677 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.964309 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:11.964327 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:11.965841 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:11.967839 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.967842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.967851 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.968065 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.968302 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.968304 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.968305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.968416 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.968753 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:11.968758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:11.968983 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 20:18:11.969121 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:18:11.969401 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":141,"threads_started":1}
I20260620 20:18:11.971199 4449 heartbeater.cc:347] Connected to a master server at 127.2.235.254:35977
I20260620 20:18:11.971320 4449 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:11.971804 4449 heartbeater.cc:511] Master 127.2.235.254:35977 requested a full tablet report, sending...
I20260620 20:18:11.972859 4105 ts_manager.cc:194] Registered new tserver with Master: 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745)
I20260620 20:18:11.972872 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4318
I20260620 20:18:11.972970 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:11.973268 2991 external_mini_cluster.cc:1428] Setting key 0f597095ac41734ea98d3bc6ed04ce65
I20260620 20:18:11.973570 4105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:48307
I20260620 20:18:11.980926 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.973288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.2.235.194:58291)
0620 20:18:11.973631 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:11.973646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:11.973775 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:11.974040 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:11.974043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:11.974246 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:11.974362 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.974369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.975398 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.975403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.976221 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:11.976229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.976802 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.976805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:11.976886 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.977366 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:11.977378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:11.977718 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:11.979566 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.979569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.979570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.979741 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.980037 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:11.980039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:11.980040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:11.980113 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:11.980585 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:11.980587 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:11.980648 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 20:18:11.980716 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:18:11.980817 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":91,"threads_started":1}
I20260620 20:18:11.981099 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:11.973329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:58291 (local address 127.2.235.254:35977)
0620 20:18:11.973470 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:11.973474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:11.973706 (+ 232us) server_negotiation.cc:407] Connection header received
0620 20:18:11.973887 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:11.973891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:11.973931 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:11.974024 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:11.974530 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.975260 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.976335 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:11.976688 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:11.976797 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:11.977841 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:11.977847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:11.977853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:11.977882 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:11.979460 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.979837 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.979843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.979847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.979938 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:11.980202 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:11.980208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:11.980212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:11.980496 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:11.980596 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:11.980814 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:18:11.980921 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:11.997582 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:35977
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:12.098012 4458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:12.098217 4458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:12.098256 4458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:12.102066 4458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:12.102113 4458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:12.102149 4458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:12.102180 4458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:12.107213 4458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:35977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:12.108007 4458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:12.108830 4458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:12.121436 4463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:12.121516 4466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:12.121492 4458 server_base.cc:1034] running on GCE node
W20260620 20:18:12.121498 4464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:12.122056 4458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:12.122622 4458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:12.123806 4458 hybrid_clock.cc:648] HybridClock initialized: now 1781986692123792 us; error 28 us; skew 500 ppm
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986692, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:12.126416 4458 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:12.127182 4458 webserver.cc:460] Webserver started at http://127.2.235.195:38375/ using document root <none> and password file <none>
I20260620 20:18:12.127714 4458 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:12.127805 4458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:12.128026 4458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:12.129974 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "61da93df0a9246f3aafb9102c0cd1e2e"
format_stamp: "Formatted at 2026-06-20 20:18:12 on dist-test-slave-8z53"
server_key: "4fbda5b191c93f6bf7d75b12e10993de"
server_key_iv: "6c1c965ae76f96ef7f243892f575dc6b"
server_key_version: "encryptionkey@0"
I20260620 20:18:12.130559 4458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "61da93df0a9246f3aafb9102c0cd1e2e"
format_stamp: "Formatted at 2026-06-20 20:18:12 on dist-test-slave-8z53"
server_key: "4fbda5b191c93f6bf7d75b12e10993de"
server_key_iv: "6c1c965ae76f96ef7f243892f575dc6b"
server_key_version: "encryptionkey@0"
I20260620 20:18:12.133987 4458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:12.136083 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:12.137161 4458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:12.137308 4458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "61da93df0a9246f3aafb9102c0cd1e2e"
format_stamp: "Formatted at 2026-06-20 20:18:12 on dist-test-slave-8z53"
server_key: "4fbda5b191c93f6bf7d75b12e10993de"
server_key_iv: "6c1c965ae76f96ef7f243892f575dc6b"
server_key_version: "encryptionkey@0"
I20260620 20:18:12.137413 4458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:12.150570 4458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:12.153261 4458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:12.153519 4458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:12.154742 4458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:12.154814 4458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:12.154874 4458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:12.154924 4458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986692, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:12.165601 4458 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:36011
I20260620 20:18:12.165735 4588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:36011 every 8 connection(s)
I20260620 20:18:12.166608 4458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:12.168534 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.155540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:49137 (local address 127.2.235.254:35977)
0620 20:18:12.155710 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.155717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.156423 (+ 706us) server_negotiation.cc:407] Connection header received
0620 20:18:12.157356 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.157360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.157416 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.157499 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:12.159076 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.160128 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.161316 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.161545 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.161658 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.164597 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:12.164600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:12.164606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:12.164636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:12.166457 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.166969 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.166974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.166978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.167076 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.167465 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.167468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.167470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.167694 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:12.167806 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.168260 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.168373 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:18:12.169193 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.155854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.2.235.195:49137)
0620 20:18:12.156273 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.156318 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.157121 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.157755 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.157764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.158245 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:12.158874 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.158897 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.160239 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.160242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.161157 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.161166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.161729 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.161736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.161991 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.162596 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:12.162619 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:12.164440 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:12.166558 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:12.166562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:12.166571 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:12.166872 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:12.167198 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:12.167203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:12.167207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:12.167367 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:12.167804 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:12.167810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:12.168025 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.168204 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.168478 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
I20260620 20:18:12.170269 4589 heartbeater.cc:347] Connected to a master server at 127.2.235.254:35977
I20260620 20:18:12.170395 4589 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:12.170913 4589 heartbeater.cc:511] Master 127.2.235.254:35977 requested a full tablet report, sending...
I20260620 20:18:12.171944 4106 ts_manager.cc:194] Registered new tserver with Master: 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011)
I20260620 20:18:12.172255 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4458
I20260620 20:18:12.172386 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:12.172668 2991 external_mini_cluster.cc:1428] Setting key 65978f9bbbe31541ddfd7138cb23b9f4
I20260620 20:18:12.172751 4106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:49137
I20260620 20:18:12.174594 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:12.180008 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.173010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:36787 (local address 127.2.235.254:35977)
0620 20:18:12.173111 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.173117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.173129 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:12.173160 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.173162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.173199 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.173334 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:12.173857 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.174499 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.175586 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.175967 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.176080 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.177138 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:12.177141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:12.177143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:12.177165 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:12.178540 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.178920 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.178924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.178925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.178999 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.179260 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.179265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.179266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.179415 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:12.179513 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.179754 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.179851 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 20:18:12.180027 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.172467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.2.235.195:36787)
0620 20:18:12.172741 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.172755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.172837 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.173448 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.173451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.173639 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:12.173745 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.173753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.174607 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.174611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.175457 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.175471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.176073 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.176076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.176139 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.176679 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:12.176692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:12.177005 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:12.178638 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:12.178642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:12.178645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:12.178825 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:12.179090 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:12.179093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:12.179095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:12.179175 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:12.179504 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:12.179507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:12.179602 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.179703 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.179828 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:12.198180 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.191384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.2.235.254:35977)
0620 20:18:12.191483 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.191486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.191496 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:18:12.191523 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.191524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.191553 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.191622 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:12.192189 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.192629 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.193442 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.193643 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.193716 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.195752 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:12.195755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:12.195757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:12.195776 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:12.196938 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.197318 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.197320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.197321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.197385 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.197608 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.197610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.197611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.197744 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:12.197836 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.198014 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.198089 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:18:12.200824 4106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:12.203642 4106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:12.217396 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.212060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:36011 (local address 127.0.0.1:54556)
0620 20:18:12.212578 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.212592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.212675 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.213675 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.213678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.213695 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.213766 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.213772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.215170 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.215176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.215959 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.215968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.216876 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.216880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.216982 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.217018 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.217122 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.217226 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":26,"thread_start_us":93,"threads_started":1}
I20260620 20:18:12.217623 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.212217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.2.235.193:40733)
0620 20:18:12.212843 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.212855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.212872 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:18:12.212928 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.212932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.213063 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.213333 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.213662 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.214453 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.216400 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.217012 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.217124 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.217299 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.217377 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.217478 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":93,"threads_started":1}
I20260620 20:18:12.217901 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.212441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.2.235.195:36011)
0620 20:18:12.212731 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.212743 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.212767 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:18:12.212834 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.212839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.213026 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.213175 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.213878 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.215055 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.216084 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.216744 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.216858 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.216993 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.217554 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.217635 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260620 20:18:12.217542 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.211793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40733 (local address 127.0.0.1:34016)
0620 20:18:12.212347 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.212361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.212439 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.213214 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.213216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.213242 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.213553 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.213558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.214671 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.214674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.216261 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.216272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.217119 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.217123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.217221 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.217281 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.217347 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.217417 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":409,"spinlock_wait_cycles":6272,"thread_start_us":91,"threads_started":1}
I20260620 20:18:12.217860 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.211386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40745 (local address 127.0.0.1:33020)
0620 20:18:12.211685 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.211732 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.211841 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.213201 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.213207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.213237 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.213388 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.213394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.214631 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.214635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.216261 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.216272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.217340 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.217345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.217434 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.217452 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.217607 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.217672 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":146,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260620 20:18:12.218576 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.211559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.2.235.194:40745)
0620 20:18:12.212738 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.212750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.212768 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:18:12.212835 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.212839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.213026 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.213175 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.213499 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.214536 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.216380 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.217142 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.218046 (+ 904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.218310 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.218360 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.218448 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":103,"threads_started":1}
I20260620 20:18:12.220676 4383 tablet_service.cc:1467] Processing CreateTablet for tablet 99d0fc3ee0a9420fbe89893507e3e0c8 (DEFAULT_TABLE table=test-table [id=f803f9402b8b417a98b1aca4c1adc3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:12.220852 4243 tablet_service.cc:1467] Processing CreateTablet for tablet 99d0fc3ee0a9420fbe89893507e3e0c8 (DEFAULT_TABLE table=test-table [id=f803f9402b8b417a98b1aca4c1adc3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:12.220851 4523 tablet_service.cc:1467] Processing CreateTablet for tablet 99d0fc3ee0a9420fbe89893507e3e0c8 (DEFAULT_TABLE table=test-table [id=f803f9402b8b417a98b1aca4c1adc3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:12.221552 4383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d0fc3ee0a9420fbe89893507e3e0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.221753 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d0fc3ee0a9420fbe89893507e3e0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.221758 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d0fc3ee0a9420fbe89893507e3e0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.227478 4612 tablet_bootstrap.cc:492] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Bootstrap starting.
I20260620 20:18:12.228891 4613 tablet_bootstrap.cc:492] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Bootstrap starting.
I20260620 20:18:12.229795 4614 tablet_bootstrap.cc:492] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Bootstrap starting.
I20260620 20:18:12.229832 4612 tablet_bootstrap.cc:654] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.230779 4612 log.cc:824] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:12.231376 4613 tablet_bootstrap.cc:654] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.231572 4614 tablet_bootstrap.cc:654] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.232146 4613 log.cc:824] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:12.232185 4614 log.cc:824] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:12.232719 4612 tablet_bootstrap.cc:492] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: No bootstrap required, opened a new log
I20260620 20:18:12.232894 4612 ts_tablet_manager.cc:1397] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:18:12.234118 4614 tablet_bootstrap.cc:492] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: No bootstrap required, opened a new log
I20260620 20:18:12.234288 4614 ts_tablet_manager.cc:1397] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:12.234637 4613 tablet_bootstrap.cc:492] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: No bootstrap required, opened a new log
I20260620 20:18:12.234814 4613 ts_tablet_manager.cc:1397] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 20:18:12.236667 4612 raft_consensus.cc:348] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.236944 4612 raft_consensus.cc:374] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.237007 4612 raft_consensus.cc:729] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b655e52ad942f6a244af35aac2bc26, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.237450 4612 consensus_queue.cc:260] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.237677 4613 raft_consensus.cc:348] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.237946 4613 raft_consensus.cc:374] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.238015 4613 raft_consensus.cc:729] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61da93df0a9246f3aafb9102c0cd1e2e, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.238168 4449 heartbeater.cc:503] Master 127.2.235.254:35977 was elected leader, sending a full tablet report...
I20260620 20:18:12.237998 4614 raft_consensus.cc:348] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.238257 4614 raft_consensus.cc:374] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.238312 4614 raft_consensus.cc:729] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856582ff2db7443daa5ac02c67a6b29a, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.238471 4612 ts_tablet_manager.cc:1428] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:18:12.238449 4613 consensus_queue.cc:260] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.238786 4614 consensus_queue.cc:260] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.239269 4589 heartbeater.cc:503] Master 127.2.235.254:35977 was elected leader, sending a full tablet report...
I20260620 20:18:12.239857 4614 ts_tablet_manager.cc:1428] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:18:12.239954 4613 ts_tablet_manager.cc:1428] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:18:12.240828 4309 heartbeater.cc:503] Master 127.2.235.254:35977 was elected leader, sending a full tablet report...
W20260620 20:18:12.270291 4310 tablet.cc:2369] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:12.309077 4620 raft_consensus.cc:482] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:12.309327 4620 raft_consensus.cc:504] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.310333 4620 leader_election.cc:288] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745), 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011)
I20260620 20:18:12.315061 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.310686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:35847 (local address 127.2.235.194:40745)
0620 20:18:12.310809 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.310814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.310900 (+ 86us) server_negotiation.cc:407] Connection header received
0620 20:18:12.311037 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.311043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.311090 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.311163 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.311465 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.312264 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.313806 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.314418 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.314536 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.314582 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.314869 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.314933 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:18:12.315061 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.310988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:41197 (local address 127.2.235.195:36011)
0620 20:18:12.311153 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.311158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.311286 (+ 128us) server_negotiation.cc:407] Connection header received
0620 20:18:12.311368 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.311370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.311421 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.311498 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.311725 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.312471 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.313806 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.314418 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.314536 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.314582 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.314869 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.314933 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:18:12.315143 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.310538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40745 (local address 127.2.235.193:35847)
0620 20:18:12.310808 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.310824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.310932 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.311197 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.311202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.311227 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.311354 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.311362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.312365 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.312368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.313664 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.313675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.314537 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.314541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.314700 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.314762 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.314837 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.314957 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":98,"thread_start_us":82,"threads_started":1}
I20260620 20:18:12.315152 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.310894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:36011 (local address 127.2.235.193:41197)
0620 20:18:12.311196 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.311210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.311289 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.311525 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.311527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.311541 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.311621 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.311625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.312562 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.312565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.313664 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.313675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.314537 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.314541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.314707 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.314767 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.314837 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.314957 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":117,"thread_start_us":61,"threads_started":1}
I20260620 20:18:12.315735 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0fc3ee0a9420fbe89893507e3e0c8" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b655e52ad942f6a244af35aac2bc26" is_pre_election: true
I20260620 20:18:12.315745 4543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0fc3ee0a9420fbe89893507e3e0c8" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" is_pre_election: true
I20260620 20:18:12.315979 4403 raft_consensus.cc:2454] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 0.
I20260620 20:18:12.316001 4543 raft_consensus.cc:2454] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 0.
I20260620 20:18:12.316424 4197 leader_election.cc:302] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69b655e52ad942f6a244af35aac2bc26, 856582ff2db7443daa5ac02c67a6b29a; no voters:
I20260620 20:18:12.316686 4620 raft_consensus.cc:2781] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:12.316752 4620 raft_consensus.cc:482] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:12.316781 4620 raft_consensus.cc:3037] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.317974 4620 raft_consensus.cc:504] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.318267 4620 leader_election.cc:288] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 election: Requested vote from peers 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745), 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011)
I20260620 20:18:12.318600 4543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0fc3ee0a9420fbe89893507e3e0c8" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61da93df0a9246f3aafb9102c0cd1e2e"
I20260620 20:18:12.318645 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d0fc3ee0a9420fbe89893507e3e0c8" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b655e52ad942f6a244af35aac2bc26"
I20260620 20:18:12.318717 4543 raft_consensus.cc:3037] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.318745 4403 raft_consensus.cc:3037] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.319936 4543 raft_consensus.cc:2454] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 1.
I20260620 20:18:12.319988 4403 raft_consensus.cc:2454] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 1.
I20260620 20:18:12.320235 4197 leader_election.cc:302] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61da93df0a9246f3aafb9102c0cd1e2e, 856582ff2db7443daa5ac02c67a6b29a; no voters:
I20260620 20:18:12.320410 4620 raft_consensus.cc:2781] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:12.320595 4620 raft_consensus.cc:686] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 1 LEADER]: Becoming Leader. State: Replica: 856582ff2db7443daa5ac02c67a6b29a, State: Running, Role: LEADER
I20260620 20:18:12.320888 4620 consensus_queue.cc:237] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } }
I20260620 20:18:12.323801 4104 catalog_manager.cc:5495] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193). New cstate: current_term: 1 leader_uuid: "856582ff2db7443daa5ac02c67a6b29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } health_report { overall_health: HEALTHY } } }
I20260620 20:18:12.339923 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.336492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.2.235.193:40733)
0620 20:18:12.336627 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.336631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.336646 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:18:12.336803 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.336806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.336874 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.336950 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:12.337166 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.337685 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.338491 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.338741 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.338841 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.339096 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.339526 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.339641 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.339738 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.339792 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:18:12.344674 4543 raft_consensus.cc:1261] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Refusing update from remote peer 856582ff2db7443daa5ac02c67a6b29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:12.344756 4403 raft_consensus.cc:1261] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Refusing update from remote peer 856582ff2db7443daa5ac02c67a6b29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:12.345340 4620 consensus_queue.cc:1035] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:12.345491 4623 consensus_queue.cc:1035] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:12.351570 4626 mvcc.cc:205] Tried to move back new op lower bound from 7299017491838115840 to 7299017491748917248. Current Snapshot: MvccSnapshot[applied={T|T < 7299017491838115840}]
I20260620 20:18:12.353744 4627 mvcc.cc:205] Tried to move back new op lower bound from 7299017491838115840 to 7299017491748917248. Current Snapshot: MvccSnapshot[applied={T|T < 7299017491838115840 or (T in {7299017491838115840})}]
I20260620 20:18:12.353967 4628 mvcc.cc:205] Tried to move back new op lower bound from 7299017491838115840 to 7299017491748917248. Current Snapshot: MvccSnapshot[applied={T|T < 7299017491838115840}]
I20260620 20:18:12.362645 4104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:12.362766 4104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:12.364979 4104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: Sending DeleteTablet for 3 replicas of tablet 99d0fc3ee0a9420fbe89893507e3e0c8
I20260620 20:18:12.365655 4383 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d0fc3ee0a9420fbe89893507e3e0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:12 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33020
I20260620 20:18:12.365710 4523 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d0fc3ee0a9420fbe89893507e3e0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:12 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:54556
I20260620 20:18:12.365821 4243 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d0fc3ee0a9420fbe89893507e3e0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:12 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:34016
I20260620 20:18:12.366086 4636 tablet_replica.cc:330] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: stopping tablet replica
I20260620 20:18:12.366254 4638 tablet_replica.cc:330] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: stopping tablet replica
I20260620 20:18:12.366356 4636 raft_consensus.cc:2229] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:12.366457 4638 raft_consensus.cc:2229] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:12.366624 4636 raft_consensus.cc:2258] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:12.366750 4638 raft_consensus.cc:2258] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:12.366705 4104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:18:12.366947 4637 tablet_replica.cc:330] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: stopping tablet replica
W20260620 20:18:12.367044 4104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:12.367205 4637 raft_consensus.cc:2229] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:12.367444 4637 raft_consensus.cc:2258] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:12.368037 4638 ts_tablet_manager.cc:1905] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:12.369122 4637 ts_tablet_manager.cc:1905] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:12.370724 4638 ts_tablet_manager.cc:1918] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:12.370836 4638 log.cc:1196] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/99d0fc3ee0a9420fbe89893507e3e0c8
I20260620 20:18:12.371217 4638 ts_tablet_manager.cc:1939] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 856582ff2db7443daa5ac02c67a6b29a: Deleting consensus metadata
I20260620 20:18:12.372032 4637 ts_tablet_manager.cc:1918] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:12.372184 4637 log.cc:1196] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/99d0fc3ee0a9420fbe89893507e3e0c8
I20260620 20:18:12.372278 4092 catalog_manager.cc:4841] TS 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193:40733): tablet 99d0fc3ee0a9420fbe89893507e3e0c8 (table test-table [id=f803f9402b8b417a98b1aca4c1adc3d7]) successfully deleted
I20260620 20:18:12.372357 4243 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea6d93232d44d028f7be8b95bba5d07 (DEFAULT_TABLE table=test-table [id=0cd15d39422647d1baa7b7dc9a2c37db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:12.372594 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea6d93232d44d028f7be8b95bba5d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.372699 4637 ts_tablet_manager.cc:1939] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 61da93df0a9246f3aafb9102c0cd1e2e: Deleting consensus metadata
I20260620 20:18:12.373528 4523 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea6d93232d44d028f7be8b95bba5d07 (DEFAULT_TABLE table=test-table [id=0cd15d39422647d1baa7b7dc9a2c37db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:12.373793 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea6d93232d44d028f7be8b95bba5d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.374636 4614 tablet_bootstrap.cc:492] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Bootstrap starting.
I20260620 20:18:12.375087 4636 ts_tablet_manager.cc:1905] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:12.375519 4614 tablet_bootstrap.cc:654] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.375944 4613 tablet_bootstrap.cc:492] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Bootstrap starting.
I20260620 20:18:12.376157 4093 catalog_manager.cc:4841] TS 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011): tablet 99d0fc3ee0a9420fbe89893507e3e0c8 (table test-table [id=f803f9402b8b417a98b1aca4c1adc3d7]) successfully deleted
I20260620 20:18:12.376901 4613 tablet_bootstrap.cc:654] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.376911 4383 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea6d93232d44d028f7be8b95bba5d07 (DEFAULT_TABLE table=test-table [id=0cd15d39422647d1baa7b7dc9a2c37db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:12.377130 4383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea6d93232d44d028f7be8b95bba5d07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.377722 4636 ts_tablet_manager.cc:1918] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:12.377799 4636 log.cc:1196] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/99d0fc3ee0a9420fbe89893507e3e0c8
I20260620 20:18:12.378057 4613 tablet_bootstrap.cc:492] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: No bootstrap required, opened a new log
I20260620 20:18:12.378079 4636 ts_tablet_manager.cc:1939] T 99d0fc3ee0a9420fbe89893507e3e0c8 P 69b655e52ad942f6a244af35aac2bc26: Deleting consensus metadata
I20260620 20:18:12.378144 4613 ts_tablet_manager.cc:1397] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:12.378535 4613 raft_consensus.cc:348] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.378630 4613 raft_consensus.cc:374] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.378662 4613 raft_consensus.cc:729] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61da93df0a9246f3aafb9102c0cd1e2e, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.378773 4613 consensus_queue.cc:260] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.379040 4613 ts_tablet_manager.cc:1428] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:12.379153 4093 catalog_manager.cc:4841] TS 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745): tablet 99d0fc3ee0a9420fbe89893507e3e0c8 (table test-table [id=f803f9402b8b417a98b1aca4c1adc3d7]) successfully deleted
I20260620 20:18:12.379716 4612 tablet_bootstrap.cc:492] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Bootstrap starting.
I20260620 20:18:12.380342 4614 tablet_bootstrap.cc:492] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: No bootstrap required, opened a new log
I20260620 20:18:12.380414 4614 ts_tablet_manager.cc:1397] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 20:18:12.380635 4612 tablet_bootstrap.cc:654] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.380801 4614 raft_consensus.cc:348] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.380900 4614 raft_consensus.cc:374] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.380931 4614 raft_consensus.cc:729] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856582ff2db7443daa5ac02c67a6b29a, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.381055 4614 consensus_queue.cc:260] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.381302 4614 ts_tablet_manager.cc:1428] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:12.382500 4612 tablet_bootstrap.cc:492] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: No bootstrap required, opened a new log
I20260620 20:18:12.382568 4612 ts_tablet_manager.cc:1397] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:18:12.382964 4612 raft_consensus.cc:348] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.383059 4612 raft_consensus.cc:374] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.383092 4612 raft_consensus.cc:729] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b655e52ad942f6a244af35aac2bc26, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.383201 4612 consensus_queue.cc:260] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.383416 4612 ts_tablet_manager.cc:1428] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:12.405047 4631 raft_consensus.cc:482] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:12.405166 4631 raft_consensus.cc:504] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.405542 4631 leader_election.cc:288] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745), 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011)
I20260620 20:18:12.405893 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea6d93232d44d028f7be8b95bba5d07" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b655e52ad942f6a244af35aac2bc26" is_pre_election: true
I20260620 20:18:12.405969 4543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea6d93232d44d028f7be8b95bba5d07" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" is_pre_election: true
I20260620 20:18:12.406025 4403 raft_consensus.cc:2454] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 0.
I20260620 20:18:12.406091 4543 raft_consensus.cc:2454] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 0.
I20260620 20:18:12.406332 4197 leader_election.cc:302] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61da93df0a9246f3aafb9102c0cd1e2e, 856582ff2db7443daa5ac02c67a6b29a; no voters:
I20260620 20:18:12.406476 4631 raft_consensus.cc:2781] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:12.406533 4631 raft_consensus.cc:482] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:12.406594 4631 raft_consensus.cc:3037] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.407407 4631 raft_consensus.cc:504] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.407697 4631 leader_election.cc:288] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 election: Requested vote from peers 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745), 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011)
I20260620 20:18:12.407968 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea6d93232d44d028f7be8b95bba5d07" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b655e52ad942f6a244af35aac2bc26"
I20260620 20:18:12.407997 4543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea6d93232d44d028f7be8b95bba5d07" candidate_uuid: "856582ff2db7443daa5ac02c67a6b29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61da93df0a9246f3aafb9102c0cd1e2e"
I20260620 20:18:12.408079 4543 raft_consensus.cc:3037] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.408087 4403 raft_consensus.cc:3037] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.408990 4403 raft_consensus.cc:2454] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 1.
I20260620 20:18:12.408990 4543 raft_consensus.cc:2454] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856582ff2db7443daa5ac02c67a6b29a in term 1.
I20260620 20:18:12.409262 4197 leader_election.cc:302] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61da93df0a9246f3aafb9102c0cd1e2e, 856582ff2db7443daa5ac02c67a6b29a; no voters:
I20260620 20:18:12.409461 4631 raft_consensus.cc:2781] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:12.409582 4631 raft_consensus.cc:686] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 1 LEADER]: Becoming Leader. State: Replica: 856582ff2db7443daa5ac02c67a6b29a, State: Running, Role: LEADER
I20260620 20:18:12.409711 4631 consensus_queue.cc:237] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.411046 4104 catalog_manager.cc:5495] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a reported cstate change: term changed from 0 to 1, leader changed from <none> to 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193). New cstate: current_term: 1 leader_uuid: "856582ff2db7443daa5ac02c67a6b29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } health_report { overall_health: UNKNOWN } } }
W20260620 20:18:12.417455 4590 tablet.cc:2369] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986691, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:12.447065 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.438680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:35977 (local address 127.0.0.1:41806)
0620 20:18:12.438934 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.438953 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.439029 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.439257 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.439259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.439724 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:12.439806 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.439812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.440785 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.440788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.441501 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.441510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.442022 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.442025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.442115 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.442691 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:12.442703 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:12.443998 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:12.445743 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:12.445750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:12.445754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:12.445969 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:12.446250 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:12.446252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:12.446253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:12.446321 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:12.446676 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:12.446680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:12.446767 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.446857 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.446951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":16,"thread_start_us":111,"threads_started":1}
I20260620 20:18:12.447201 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.438711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.2.235.254:35977)
0620 20:18:12.438822 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.438827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.439013 (+ 186us) server_negotiation.cc:407] Connection header received
0620 20:18:12.439111 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.439113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.439155 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.439241 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:12.439905 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.440661 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.441601 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.441919 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.442014 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.444130 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:12.444134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:12.444137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:12.444162 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:12.445649 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.446066 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.446069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.446071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.446153 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.446397 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.446400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.446401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.446580 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:12.446666 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.446928 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.447027 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":38}
I20260620 20:18:12.450600 4104 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:41806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:18:12.451395 4104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:12.455988 4243 tablet_service.cc:1467] Processing CreateTablet for tablet d81aea9fcc4f45b6b5052e02351905e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ef011877e814bc1832e6451abe50d87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:12.456120 4383 tablet_service.cc:1467] Processing CreateTablet for tablet d81aea9fcc4f45b6b5052e02351905e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ef011877e814bc1832e6451abe50d87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:12.456348 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81aea9fcc4f45b6b5052e02351905e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.456446 4383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81aea9fcc4f45b6b5052e02351905e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.457630 4523 tablet_service.cc:1467] Processing CreateTablet for tablet d81aea9fcc4f45b6b5052e02351905e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ef011877e814bc1832e6451abe50d87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:12.457953 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81aea9fcc4f45b6b5052e02351905e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:12.459101 4614 tablet_bootstrap.cc:492] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: Bootstrap starting.
I20260620 20:18:12.459126 4612 tablet_bootstrap.cc:492] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: Bootstrap starting.
I20260620 20:18:12.459949 4614 tablet_bootstrap.cc:654] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.459971 4612 tablet_bootstrap.cc:654] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.460842 4613 tablet_bootstrap.cc:492] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: Bootstrap starting.
I20260620 20:18:12.461424 4612 tablet_bootstrap.cc:492] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: No bootstrap required, opened a new log
I20260620 20:18:12.461500 4612 ts_tablet_manager.cc:1397] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:12.461833 4613 tablet_bootstrap.cc:654] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:12.461922 4612 raft_consensus.cc:348] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.462041 4612 raft_consensus.cc:374] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.462069 4612 raft_consensus.cc:729] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b655e52ad942f6a244af35aac2bc26, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.462185 4612 consensus_queue.cc:260] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.462445 4612 ts_tablet_manager.cc:1428] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:18:12.462713 4618 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.462849 4618 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:12.463640 4614 tablet_bootstrap.cc:492] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: No bootstrap required, opened a new log
I20260620 20:18:12.463716 4614 ts_tablet_manager.cc:1397] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 20:18:12.463908 4613 tablet_bootstrap.cc:492] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: No bootstrap required, opened a new log
I20260620 20:18:12.463975 4613 ts_tablet_manager.cc:1397] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:12.464160 4614 raft_consensus.cc:348] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.464251 4614 raft_consensus.cc:374] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.464285 4614 raft_consensus.cc:729] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856582ff2db7443daa5ac02c67a6b29a, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.464398 4614 consensus_queue.cc:260] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.464389 4613 raft_consensus.cc:348] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.464479 4613 raft_consensus.cc:374] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:12.464514 4613 raft_consensus.cc:729] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61da93df0a9246f3aafb9102c0cd1e2e, State: Initialized, Role: FOLLOWER
I20260620 20:18:12.464623 4613 consensus_queue.cc:260] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.464682 4614 ts_tablet_manager.cc:1428] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:12.464874 4613 ts_tablet_manager.cc:1428] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:12.465055 4631 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.465167 4631 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:12.465370 4619 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.465504 4619 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 20:18:12.467929 4450 tablet.cc:2369] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:12.582171 4619 raft_consensus.cc:482] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:12.582330 4619 raft_consensus.cc:504] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.583386 4619 leader_election.cc:288] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193:40733), 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745)
I20260620 20:18:12.588167 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.583777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:38995 (local address 127.2.235.194:40745)
0620 20:18:12.583927 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.583932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.584017 (+ 85us) server_negotiation.cc:407] Connection header received
0620 20:18:12.584133 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.584136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.584183 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.584262 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.584541 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.585325 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.586917 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.587478 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.587601 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.587657 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.587971 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.588039 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:18:12.588165 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.583625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:45857 (local address 127.2.235.193:40733)
0620 20:18:12.583737 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.583741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.583876 (+ 135us) server_negotiation.cc:407] Connection header received
0620 20:18:12.583948 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.583953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.584012 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.584109 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:12.584815 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.585539 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.586917 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.587478 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.587602 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.587657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.587971 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.588041 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:18:12.588217 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.583566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40733 (local address 127.2.235.195:45857)
0620 20:18:12.583723 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.583737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.583828 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.584144 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.584146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.584159 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.584694 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.584701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.585646 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.585649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.586794 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.586805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.587603 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.587606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.587785 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.587851 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.587931 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.588028 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":150}
I20260620 20:18:12.588235 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.583586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40745 (local address 127.2.235.195:38995)
0620 20:18:12.583940 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.583952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.584035 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.584298 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.584300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.584312 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:12.584417 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.584424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.585436 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.585438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.586794 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.586805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.587607 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.587609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.587807 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.587848 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.587931 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.588028 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":104,"thread_start_us":109,"threads_started":1}
I20260620 20:18:12.588713 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81aea9fcc4f45b6b5052e02351905e7" candidate_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b655e52ad942f6a244af35aac2bc26" is_pre_election: true
I20260620 20:18:12.588886 4403 raft_consensus.cc:2454] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61da93df0a9246f3aafb9102c0cd1e2e in term 0.
I20260620 20:18:12.588980 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81aea9fcc4f45b6b5052e02351905e7" candidate_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856582ff2db7443daa5ac02c67a6b29a" is_pre_election: true
I20260620 20:18:12.589268 4263 raft_consensus.cc:2454] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61da93df0a9246f3aafb9102c0cd1e2e in term 0.
I20260620 20:18:12.589308 4477 leader_election.cc:302] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61da93df0a9246f3aafb9102c0cd1e2e, 69b655e52ad942f6a244af35aac2bc26; no voters:
I20260620 20:18:12.589583 4619 raft_consensus.cc:2781] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:12.589641 4619 raft_consensus.cc:482] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:12.589670 4619 raft_consensus.cc:3037] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.590754 4619 raft_consensus.cc:504] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.591142 4619 leader_election.cc:288] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [CANDIDATE]: Term 1 election: Requested vote from peers 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193:40733), 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745)
I20260620 20:18:12.591429 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81aea9fcc4f45b6b5052e02351905e7" candidate_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69b655e52ad942f6a244af35aac2bc26"
I20260620 20:18:12.591533 4403 raft_consensus.cc:3037] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.592391 4403 raft_consensus.cc:2454] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61da93df0a9246f3aafb9102c0cd1e2e in term 1.
I20260620 20:18:12.592660 4263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81aea9fcc4f45b6b5052e02351905e7" candidate_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856582ff2db7443daa5ac02c67a6b29a"
I20260620 20:18:12.592711 4477 leader_election.cc:302] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61da93df0a9246f3aafb9102c0cd1e2e, 69b655e52ad942f6a244af35aac2bc26; no voters:
I20260620 20:18:12.592788 4263 raft_consensus.cc:3037] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:12.592916 4619 raft_consensus.cc:2781] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:12.593142 4619 raft_consensus.cc:686] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 LEADER]: Becoming Leader. State: Replica: 61da93df0a9246f3aafb9102c0cd1e2e, State: Running, Role: LEADER
I20260620 20:18:12.593427 4619 consensus_queue.cc:237] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } }
I20260620 20:18:12.594039 4263 raft_consensus.cc:2454] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61da93df0a9246f3aafb9102c0cd1e2e in term 1.
I20260620 20:18:12.594745 4653 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: TxnStatusTablet state changed. Reason: New leader 61da93df0a9246f3aafb9102c0cd1e2e. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.594864 4653 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:12.595155 4655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:18:12.595320 4655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:18:12.595734 4104 catalog_manager.cc:5495] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195). New cstate: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } health_report { overall_health: HEALTHY } } }
I20260620 20:18:12.619997 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.616353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:36011 (local address 127.0.0.1:54572)
0620 20:18:12.616490 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.616504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.616586 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.616886 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.616888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.616893 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:12.616972 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.616977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.617838 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.617840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.618370 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.618376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.618941 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.618944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.619006 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.619045 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.619756 (+ 711us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.619766 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.619833 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.619896 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 20:18:12.620023 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.616424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.2.235.195:36011)
0620 20:18:12.616526 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.616531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.616543 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:12.616714 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.616717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.616772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.616875 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:12.617070 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.617749 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.618504 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.618841 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.618937 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.619165 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.619645 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.619766 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.619847 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.619896 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:12.622650 4403 raft_consensus.cc:1261] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Refusing update from remote peer 61da93df0a9246f3aafb9102c0cd1e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:12.622772 4263 raft_consensus.cc:1261] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Refusing update from remote peer 61da93df0a9246f3aafb9102c0cd1e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:12.623080 4653 consensus_queue.cc:1035] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:12.623238 4619 consensus_queue.cc:1035] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:12.624555 4618 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: TxnStatusTablet state changed. Reason: New leader 61da93df0a9246f3aafb9102c0cd1e2e. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.624668 4618 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:12.626093 4631 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: TxnStatusTablet state changed. Reason: New leader 61da93df0a9246f3aafb9102c0cd1e2e. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.626202 4631 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:12.626348 4653 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.626449 4653 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:12.626685 4618 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.626765 4618 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 69b655e52ad942f6a244af35aac2bc26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:12.630357 4653 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.630445 4653 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 61da93df0a9246f3aafb9102c0cd1e2e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:12.631644 4631 tablet_replica.cc:439] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856582ff2db7443daa5ac02c67a6b29a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40733 } } peers { permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 } } peers { permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 } } }
I20260620 20:18:12.631731 4631 tablet_replica.cc:442] T d81aea9fcc4f45b6b5052e02351905e7 P 856582ff2db7443daa5ac02c67a6b29a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:12.641746 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.637172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:52919 (local address 127.2.235.195:36011)
0620 20:18:12.637301 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.637305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.638792 (+ 1487us) server_negotiation.cc:407] Connection header received
0620 20:18:12.638903 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.638905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.638938 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.639019 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:12.639278 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.639796 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.640727 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.640929 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.641022 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.641292 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.641350 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.641426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.641600 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.641641 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:18:12.641779 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.637107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:36011 (local address 127.2.235.193:52919)
0620 20:18:12.638706 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.638722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.638823 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.639056 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.639059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.639067 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:12.639171 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.639187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.639888 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.639893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.640604 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.640613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.641024 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.641027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.641110 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.641152 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.641461 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.641468 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.641586 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.641664 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"thread_start_us":96,"threads_started":1}
I20260620 20:18:12.647321 4403 raft_consensus.cc:1261] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Refusing update from remote peer 856582ff2db7443daa5ac02c67a6b29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:12.647356 4543 raft_consensus.cc:1261] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Refusing update from remote peer 856582ff2db7443daa5ac02c67a6b29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:12.647708 4620 consensus_queue.cc:1035] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61da93df0a9246f3aafb9102c0cd1e2e" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:12.647820 4631 consensus_queue.cc:1035] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69b655e52ad942f6a244af35aac2bc26" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:12.669171 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.665982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:40471 (local address 127.2.235.193:40733)
0620 20:18:12.666069 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.666074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.666113 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:18:12.666228 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.666230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.666261 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.666313 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:12.666523 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.667010 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.668096 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.668422 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.668523 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.668805 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.668864 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.668914 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.669049 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.669088 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 20:18:12.669241 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.665906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40733 (local address 127.2.235.195:40471)
0620 20:18:12.666053 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:12.666066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:12.666151 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:12.666338 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:12.666340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:12.666346 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:12.666442 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.666447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.667128 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.667130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.667999 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:12.668008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.668520 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.668523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:12.668590 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.668661 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:12.668944 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:12.668950 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:18:12.669030 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:18:12.669127 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 20:18:12.687561 4104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:12.687709 4104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:12.689642 4104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f43ac54dc544e41ac2d473af15eb09a: Sending DeleteTablet for 3 replicas of tablet 5ea6d93232d44d028f7be8b95bba5d07
I20260620 20:18:12.690184 4243 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea6d93232d44d028f7be8b95bba5d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:12 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:34016
I20260620 20:18:12.690394 4638 tablet_replica.cc:330] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: stopping tablet replica
I20260620 20:18:12.690539 4638 raft_consensus.cc:2229] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:12.690922 4638 raft_consensus.cc:2258] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:12.691823 4638 ts_tablet_manager.cc:1905] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:12.693467 4383 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea6d93232d44d028f7be8b95bba5d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:12 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33020
I20260620 20:18:12.693605 4636 tablet_replica.cc:330] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: stopping tablet replica
I20260620 20:18:12.693694 4636 raft_consensus.cc:2229] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:12.693787 4636 raft_consensus.cc:2258] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:12.694295 4523 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea6d93232d44d028f7be8b95bba5d07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:12 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:54556
I20260620 20:18:12.694468 4637 tablet_replica.cc:330] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: stopping tablet replica
I20260620 20:18:12.694579 4637 raft_consensus.cc:2229] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:12.694674 4637 raft_consensus.cc:2258] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:12.694779 4638 ts_tablet_manager.cc:1918] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:12.694841 4638 log.cc:1196] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/5ea6d93232d44d028f7be8b95bba5d07
I20260620 20:18:12.694954 4636 ts_tablet_manager.cc:1905] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:12.695107 4638 ts_tablet_manager.cc:1939] T 5ea6d93232d44d028f7be8b95bba5d07 P 856582ff2db7443daa5ac02c67a6b29a: Deleting consensus metadata
I20260620 20:18:12.695343 4637 ts_tablet_manager.cc:1905] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:12.697141 4636 ts_tablet_manager.cc:1918] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:12.697189 4636 log.cc:1196] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/5ea6d93232d44d028f7be8b95bba5d07
I20260620 20:18:12.697430 4636 ts_tablet_manager.cc:1939] T 5ea6d93232d44d028f7be8b95bba5d07 P 69b655e52ad942f6a244af35aac2bc26: Deleting consensus metadata
I20260620 20:18:12.697916 4637 ts_tablet_manager.cc:1918] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:12.697934 4093 catalog_manager.cc:4841] TS 69b655e52ad942f6a244af35aac2bc26 (127.2.235.194:40745): tablet 5ea6d93232d44d028f7be8b95bba5d07 (table test-table [id=0cd15d39422647d1baa7b7dc9a2c37db]) successfully deleted
I20260620 20:18:12.697995 4637 log.cc:1196] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/5ea6d93232d44d028f7be8b95bba5d07
I20260620 20:18:12.698241 4637 ts_tablet_manager.cc:1939] T 5ea6d93232d44d028f7be8b95bba5d07 P 61da93df0a9246f3aafb9102c0cd1e2e: Deleting consensus metadata
I20260620 20:18:12.698768 4093 catalog_manager.cc:4841] TS 61da93df0a9246f3aafb9102c0cd1e2e (127.2.235.195:36011): tablet 5ea6d93232d44d028f7be8b95bba5d07 (table test-table [id=0cd15d39422647d1baa7b7dc9a2c37db]) successfully deleted
I20260620 20:18:12.698841 4092 catalog_manager.cc:4841] TS 856582ff2db7443daa5ac02c67a6b29a (127.2.235.193:40733): tablet 5ea6d93232d44d028f7be8b95bba5d07 (table test-table [id=0cd15d39422647d1baa7b7dc9a2c37db]) successfully deleted
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:12.710618 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.691335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.2.235.193:40733)
0620 20:18:12.695612 (+ 4277us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.695616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.701507 (+ 5891us) server_negotiation.cc:407] Connection header received
0620 20:18:12.701648 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.701651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.701691 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.701773 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:12.702384 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.702833 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.703589 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.703804 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.703919 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.707033 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:12.707045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:12.707085 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:12.707110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:12.708962 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.709426 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.709432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.709434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.709522 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.709779 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.709782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.709784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.710047 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:12.710240 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.710364 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.710473 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4213}
W20260620 20:18:12.711021 4303 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34048
I20260620 20:18:12.720134 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:12.712709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.2.235.254:35977)
0620 20:18:12.712858 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:12.712862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:12.712912 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:18:12.713013 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:12.713016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:12.713058 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:12.713138 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:12.713743 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.714562 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.715283 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:12.715588 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:12.715684 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:12.716834 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:12.716839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:12.716842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:12.716868 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:12.718507 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.718973 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.718976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.718978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.719056 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:12.719340 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:12.719345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:12.719346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:12.719522 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:12.719644 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:12.719852 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:18:12.719976 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 20:18:12.720602 4104 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41812
I20260620 20:18:12.721815 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4178
I20260620 20:18:12.729311 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4318
I20260620 20:18:12.735828 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4458
I20260620 20:18:12.742102 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4074
2026-06-20T20:18:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2467 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4673](info): set up 2 sockets
Jun 20 20:18:12 dist-test-slave-8z53 krb5kdc[4673](info): commencing operation
krb5kdc: starting...
W20260620 20:18:13.728725 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:13 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:13.833405 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40413
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:40413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:13.939677 4686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:13.939886 4686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:13.939954 4686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:13.943763 4686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:13.943806 4686 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:18:13.943822 4686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:13.943838 4686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:13.943853 4686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:13.943867 4686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:13.948736 4686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:40413
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:13.949707 4686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:13.950575 4686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:13.958047 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:13.958184 4686 server_base.cc:1034] running on GCE node
W20260620 20:18:13.958046 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:13.958170 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:13.958781 4686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:13.959636 4686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:13.960786 4686 hybrid_clock.cc:648] HybridClock initialized: now 1781986693960770 us; error 37 us; skew 500 ppm
Jun 20 20:18:13 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:13.963250 4686 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:13.964048 4686 webserver.cc:460] Webserver started at http://127.2.235.254:40471/ using document root <none> and password file <none>
I20260620 20:18:13.964552 4686 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:13.964636 4686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:13.964843 4686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:13.966635 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "7eca5971138e40209e364e109f1cba86"
format_stamp: "Formatted at 2026-06-20 20:18:13 on dist-test-slave-8z53"
server_key: "5e09705672f7e88694140be8032f1b6c"
server_key_iv: "2ba9be3709cf08b886d401992161386a"
server_key_version: "encryptionkey@0"
I20260620 20:18:13.967167 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "7eca5971138e40209e364e109f1cba86"
format_stamp: "Formatted at 2026-06-20 20:18:13 on dist-test-slave-8z53"
server_key: "5e09705672f7e88694140be8032f1b6c"
server_key_iv: "2ba9be3709cf08b886d401992161386a"
server_key_version: "encryptionkey@0"
I20260620 20:18:13.970376 4686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:13.972446 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:13.973613 4686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:13.973737 4686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "7eca5971138e40209e364e109f1cba86"
format_stamp: "Formatted at 2026-06-20 20:18:13 on dist-test-slave-8z53"
server_key: "5e09705672f7e88694140be8032f1b6c"
server_key_iv: "2ba9be3709cf08b886d401992161386a"
server_key_version: "encryptionkey@0"
I20260620 20:18:13.973835 4686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:13.991226 4686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:13.993579 4686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:13.993753 4686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:14.000671 4686 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:40413
I20260620 20:18:14.000710 4763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:40413 every 8 connection(s)
I20260620 20:18:14.001441 4686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:14.003724 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.008023 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4686
I20260620 20:18:14.008142 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:14.008401 2991 external_mini_cluster.cc:1428] Setting key 74235a7c58ddc2acbe3e21c229053146
I20260620 20:18:14.008834 4764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Bootstrap starting.
I20260620 20:18:14.010998 4764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.011639 4764 log.cc:824] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:14.013139 4764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: No bootstrap required, opened a new log
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:14.016191 4764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eca5971138e40209e364e109f1cba86" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40413 } }
I20260620 20:18:14.016402 4764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.016466 4764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eca5971138e40209e364e109f1cba86, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.016865 4764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eca5971138e40209e364e109f1cba86" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40413 } }
I20260620 20:18:14.017017 4764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:14.017092 4764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:14.017194 4764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.018023 4764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eca5971138e40209e364e109f1cba86" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40413 } }
I20260620 20:18:14.018308 4764 leader_election.cc:302] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eca5971138e40209e364e109f1cba86; no voters:
I20260620 20:18:14.018600 4764 leader_election.cc:288] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:14.018679 4769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:14.018839 4769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [term 1 LEADER]: Becoming Leader. State: Replica: 7eca5971138e40209e364e109f1cba86, State: Running, Role: LEADER
I20260620 20:18:14.019140 4769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eca5971138e40209e364e109f1cba86" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40413 } }
I20260620 20:18:14.019690 4764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:14.020984 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.009681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.2.235.254:40413)
0620 20:18:14.010063 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.010075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.010105 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:18:14.010630 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.010644 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.010915 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.011197 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.011924 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.012763 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.013644 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.014084 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.014266 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.016431 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.016436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.016452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.016482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.018205 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.018638 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.018643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.018648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.018785 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.019062 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.019066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.019068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.019485 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.019596 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.019987 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.020247 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260620 20:18:14.022341 4770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eca5971138e40209e364e109f1cba86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eca5971138e40209e364e109f1cba86" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40413 } } }
I20260620 20:18:14.022452 4770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [sys.catalog]: This master's current role is: LEADER
W20260620 20:18:14.024044 4783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:14.024121 4783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:14.024247 4785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:14.024284 4771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eca5971138e40209e364e109f1cba86. Latest consensus state: current_term: 1 leader_uuid: "7eca5971138e40209e364e109f1cba86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eca5971138e40209e364e109f1cba86" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40413 } } }
I20260620 20:18:14.024385 4771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:14.024936 4785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:14.029768 4785 catalog_manager.cc:1347] Generated new cluster ID: 664e742565dd4170a4e8ee1c25a18638
I20260620 20:18:14.029825 4785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:14.039901 4785 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:14.040530 4785 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:14.045024 4785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Generated new TSK 0
I20260620 20:18:14.045610 4785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:14.072230 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40413
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:14.168390 4790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:14.168609 4790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:14.168678 4790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:14.172015 4790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:14.172086 4790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:14.172161 4790 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:18:14.172196 4790 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:14.172248 4790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:14.176894 4790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:14.177772 4790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:14.178627 4790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:14.187062 4796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:14.187071 4798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:14.187105 4795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:14.187237 4790 server_base.cc:1034] running on GCE node
I20260620 20:18:14.187682 4790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:14.188138 4790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:14.189347 4790 hybrid_clock.cc:648] HybridClock initialized: now 1781986694189319 us; error 35 us; skew 500 ppm
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:14.191927 4790 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:14.192677 4790 webserver.cc:460] Webserver started at http://127.2.235.193:42495/ using document root <none> and password file <none>
I20260620 20:18:14.193107 4790 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:14.193163 4790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:14.193363 4790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:14.195154 4790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "6fae93d0542a4e6db3acbd0c74016e81"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "59d53379da6dcc4e72d58980c0ee9a59"
server_key_iv: "84688f3ef59de357342a6ed6253b8b59"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.195595 4790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "6fae93d0542a4e6db3acbd0c74016e81"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "59d53379da6dcc4e72d58980c0ee9a59"
server_key_iv: "84688f3ef59de357342a6ed6253b8b59"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.198637 4790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:14.200495 4805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:14.201660 4790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:14.201774 4790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "6fae93d0542a4e6db3acbd0c74016e81"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "59d53379da6dcc4e72d58980c0ee9a59"
server_key_iv: "84688f3ef59de357342a6ed6253b8b59"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.201864 4790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:14.222314 4790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:14.224756 4790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:14.224931 4790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:14.226224 4790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:14.226295 4790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:14.226357 4790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:14.226399 4790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:14.236675 4790 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:39785
I20260620 20:18:14.236681 4920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:39785 every 8 connection(s)
I20260620 20:18:14.237725 4790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:14.239519 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.226956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:56809 (local address 127.2.235.254:40413)
0620 20:18:14.227094 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.227098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.227941 (+ 843us) server_negotiation.cc:407] Connection header received
0620 20:18:14.228899 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.228903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.228965 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.229024 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.230368 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.231214 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.232301 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.232651 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.232769 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.235944 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.235947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.235949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.235972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.237698 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.238220 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.238223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.238225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.238307 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.238620 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.238623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.238624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.238790 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.238865 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.239301 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.239404 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:18:14.240121 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.227401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.2.235.193:56809)
0620 20:18:14.227816 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.227850 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.228658 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.229443 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.229449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.229794 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.230194 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.230206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.231318 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.231320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.232176 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.232185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.232802 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.232806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.232955 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.233519 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.233542 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.235776 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.237822 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.237829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.237843 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.238085 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.238400 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.238402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.238404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.238527 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.238875 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.238879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.239044 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.239182 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.239394 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260620 20:18:14.241541 4921 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40413
I20260620 20:18:14.241704 4921 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:14.242209 4921 heartbeater.cc:511] Master 127.2.235.254:40413 requested a full tablet report, sending...
I20260620 20:18:14.243780 4717 ts_manager.cc:194] Registered new tserver with Master: 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785)
I20260620 20:18:14.245168 4717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:56809
I20260620 20:18:14.247269 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4790
I20260620 20:18:14.247427 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:14.247792 2991 external_mini_cluster.cc:1428] Setting key 73ff1953f047e66458ffa3aaeac4b073
I20260620 20:18:14.250873 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.243431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.2.235.193:39481)
0620 20:18:14.243701 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.243714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.243804 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.244081 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.244084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.244268 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.244371 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.244378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.245223 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.245227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.245957 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.245969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.246495 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.246499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.246596 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.247095 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.247114 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.247587 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.249378 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.249384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.249387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.249586 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.249867 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.249871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.249873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.249962 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.250331 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.250334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.250452 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.250564 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.250692 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260620 20:18:14.250875 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.243535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:39481 (local address 127.2.235.254:40413)
0620 20:18:14.243664 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.243667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.243783 (+ 116us) server_negotiation.cc:407] Connection header received
0620 20:18:14.243918 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.243927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.243964 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.244039 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.244477 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.245065 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.246127 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.246354 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.246493 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.247722 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.247725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.247726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.247744 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.249234 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.249686 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.249691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.249693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.249775 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.250060 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.250063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.250065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.250229 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.250329 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.250622 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.250730 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:14.270277 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40413
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:14.371945 4930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:14.372198 4930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:14.372275 4930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:14.376187 4930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:14.376273 4930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:14.376340 4930 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:18:14.376386 4930 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:14.376475 4930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:14.381225 4930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:14.381997 4930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:14.382783 4930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:14.391776 4935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:14.391839 4936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:14.391924 4930 server_base.cc:1034] running on GCE node
W20260620 20:18:14.391794 4938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:14.392361 4930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:14.392935 4930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:14.394088 4930 hybrid_clock.cc:648] HybridClock initialized: now 1781986694394071 us; error 27 us; skew 500 ppm
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:14.401283 4930 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:14.401948 4930 webserver.cc:460] Webserver started at http://127.2.235.194:40559/ using document root <none> and password file <none>
I20260620 20:18:14.402400 4930 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:14.402449 4930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:14.402645 4930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:14.404431 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "91db6e06089347b6a6fbc888a3693f1f"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "924e60f0dd704c31af6b1d10383dab51"
server_key_iv: "1b915b7d963c4c30456375230194b72d"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.404943 4930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "91db6e06089347b6a6fbc888a3693f1f"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "924e60f0dd704c31af6b1d10383dab51"
server_key_iv: "1b915b7d963c4c30456375230194b72d"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.408150 4930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:14.410220 4945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:14.411269 4930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:14.411348 4930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "91db6e06089347b6a6fbc888a3693f1f"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "924e60f0dd704c31af6b1d10383dab51"
server_key_iv: "1b915b7d963c4c30456375230194b72d"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.411458 4930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:14.425185 4930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:14.427775 4930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:14.427937 4930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:14.429188 4930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:14.429281 4930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:14.429352 4930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:14.429387 4930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:14.439891 4930 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:42465
I20260620 20:18:14.440039 5060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:42465 every 8 connection(s)
I20260620 20:18:14.440807 4930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:14.442555 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.429991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:47913 (local address 127.2.235.254:40413)
0620 20:18:14.430117 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.430120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.430989 (+ 869us) server_negotiation.cc:407] Connection header received
0620 20:18:14.431857 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.431864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.431929 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.432026 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.433568 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.434167 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.435661 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.435922 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.436040 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.439272 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.439275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.439277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.439300 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.440773 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.441261 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.441267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.441271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.441359 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.441707 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.441711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.441715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.441883 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.441953 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.442335 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.442421 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:14.443233 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.430430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.2.235.194:47913)
0620 20:18:14.430858 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.430894 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.431627 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.432312 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.432319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.432794 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.433424 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.433439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.434288 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.434291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.435547 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.435555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.436085 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.436090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.436248 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.436765 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.436795 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.439133 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.440896 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.440901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.440912 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.441144 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.441448 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.441452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.441453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.441612 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.441963 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.441967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.442138 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.442293 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.442514 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":134,"threads_started":1}
I20260620 20:18:14.444238 5061 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40413
I20260620 20:18:14.444360 5061 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:14.444667 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 4930
I20260620 20:18:14.444787 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:14.444835 5061 heartbeater.cc:511] Master 127.2.235.254:40413 requested a full tablet report, sending...
I20260620 20:18:14.445153 2991 external_mini_cluster.cc:1428] Setting key b8644adaf75a661b8541373a1217817b
I20260620 20:18:14.445863 4718 ts_manager.cc:194] Registered new tserver with Master: 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465)
I20260620 20:18:14.446379 4718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:47913
I20260620 20:18:14.454236 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.447909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.2.235.194:52465)
0620 20:18:14.448138 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.448148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.448220 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.448410 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.448411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.448536 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.448611 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.448614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.449322 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.449325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.449808 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.449814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.450377 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.450379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.450436 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.450871 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.450882 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.451179 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.452898 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.452900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.452901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.453056 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.453360 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.453361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.453363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.453426 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.453854 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.453856 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.453920 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.454010 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.454119 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
I20260620 20:18:14.454439 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.446353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:52465 (local address 127.2.235.254:40413)
0620 20:18:14.446432 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.446436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.448219 (+ 1783us) server_negotiation.cc:407] Connection header received
0620 20:18:14.448275 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.448279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.448321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.448394 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.448701 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.449184 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.449907 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.450276 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.450381 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.451311 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.451317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.451321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.451350 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.452796 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.453153 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.453159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.453163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.453243 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.453510 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.453515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.453517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.453769 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.453866 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.454112 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.454226 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:14.470079 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40413
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:14.572980 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:14.573207 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:14.573315 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:14.577286 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:14.577339 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:14.577414 5070 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:18:14.577468 5070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:14.577531 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:14.583034 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:14.584035 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:14.585135 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:14.594251 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:14.594219 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:14.594228 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:14.594507 5070 server_base.cc:1034] running on GCE node
I20260620 20:18:14.594879 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:14.595456 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:14.596623 5070 hybrid_clock.cc:648] HybridClock initialized: now 1781986694596610 us; error 23 us; skew 500 ppm
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:14.599390 5070 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:14.600162 5070 webserver.cc:460] Webserver started at http://127.2.235.195:43921/ using document root <none> and password file <none>
I20260620 20:18:14.600646 5070 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:14.600708 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:14.600862 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:14.602783 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "4a18f0089fdb459799a0ec663eb81796"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "9e5c3528541567274227c3aa6e191ecc"
server_key_iv: "b6b1fc561a1f83a662588989154b5289"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.603323 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "4a18f0089fdb459799a0ec663eb81796"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "9e5c3528541567274227c3aa6e191ecc"
server_key_iv: "b6b1fc561a1f83a662588989154b5289"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.606889 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:18:14.609174 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:14.610466 5070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:18:14.610607 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "4a18f0089fdb459799a0ec663eb81796"
format_stamp: "Formatted at 2026-06-20 20:18:14 on dist-test-slave-8z53"
server_key: "9e5c3528541567274227c3aa6e191ecc"
server_key_iv: "b6b1fc561a1f83a662588989154b5289"
server_key_version: "encryptionkey@0"
I20260620 20:18:14.610728 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:14.625850 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:14.628499 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:14.628705 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:14.630244 5070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:14.630322 5070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:14.630378 5070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:14.630403 5070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:14.642266 5070 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:45271
I20260620 20:18:14.642375 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:45271 every 8 connection(s)
I20260620 20:18:14.643307 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:14.644655 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.630864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:56917 (local address 127.2.235.254:40413)
0620 20:18:14.630990 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.630993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.632553 (+ 1560us) server_negotiation.cc:407] Connection header received
0620 20:18:14.633636 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.633640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.633701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.633793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.635357 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.635879 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.637651 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.637940 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.638063 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.640732 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.640736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.640739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.640762 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.642478 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.642942 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.642949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.642954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.643053 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.643392 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.643396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.643398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.643645 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.643764 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.644387 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.644530 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:18:14.644734 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 5070
I20260620 20:18:14.644855 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:14.645169 2991 external_mini_cluster.cc:1428] Setting key b4761f027e3f4d0d680de980443334e6
I20260620 20:18:14.645587 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.631825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.2.235.195:56917)
0620 20:18:14.632307 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.632346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.633432 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.634002 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.634015 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.634497 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.635182 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.635207 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.635994 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.635999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.637488 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.637503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.638095 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.638100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.638271 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.638789 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.638804 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.640582 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.642591 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.642595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.642604 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.642835 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.643148 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.643150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.643152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.643280 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.643825 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.643831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.644139 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.644323 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.644643 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260620 20:18:14.646919 5201 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40413
I20260620 20:18:14.647064 5201 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:14.647588 5201 heartbeater.cc:511] Master 127.2.235.254:40413 requested a full tablet report, sending...
I20260620 20:18:14.648554 4717 ts_manager.cc:194] Registered new tserver with Master: 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:14.649224 4717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:56917
I20260620 20:18:14.655326 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.648906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.2.235.195:46747)
0620 20:18:14.649236 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.649246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.649366 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.649611 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.649614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.649794 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.649901 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.649924 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.650630 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.650633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.651320 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.651328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.651738 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.651741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.651811 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.652283 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.652305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.652581 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.654120 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.654122 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.654123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.654295 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.654520 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.654522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.654523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.654592 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.654918 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.654921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.654998 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.655102 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.655204 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
I20260620 20:18:14.655369 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.648968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:46747 (local address 127.2.235.254:40413)
0620 20:18:14.649100 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.649105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.649335 (+ 230us) server_negotiation.cc:407] Connection header received
0620 20:18:14.649464 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.649466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.649508 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.649586 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.650042 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.650497 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.651423 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.651620 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.651701 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.652692 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.652695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.652697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.652714 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.654002 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.654375 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.654377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.654379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.654445 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.654666 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.654668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.654669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.654808 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.654880 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.655189 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.655273 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:18:14.658862 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:14.683063 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.675168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.2.235.254:40413)
0620 20:18:14.675296 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.675305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.675318 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:14.675355 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.675357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.675400 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.675483 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.676175 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.677064 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.678006 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.678195 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.678270 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.680486 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.680489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.680491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.680510 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.681681 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.682050 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.682053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.682054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.682135 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.682375 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.682377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.682378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.682518 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.682609 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.682797 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.682872 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:18:14.685570 4717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:14.688369 4717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:14.702358 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.696690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:45271 (local address 127.0.0.1:33292)
0620 20:18:14.697738 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.697752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.697845 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.698391 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.698393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.698404 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:14.698525 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.698534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.699723 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.699725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.700980 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.700990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.701842 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.701950 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.701978 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.702039 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.702207 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":33,"thread_start_us":94,"threads_started":1}
I20260620 20:18:14.702366 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.697388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.2.235.194:42465)
0620 20:18:14.697749 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.697760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.697781 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:18:14.697849 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.697854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.698043 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.698180 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:14.698603 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.699998 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.700927 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.701659 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701758 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.701897 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.702128 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.702214 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260620 20:18:14.702363 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.696180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:42465 (local address 127.0.0.1:59568)
0620 20:18:14.697174 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.697187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.697310 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.698333 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.698340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.698366 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:14.698473 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.698480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.700093 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.700096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.700820 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.700829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.701783 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.701914 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.701976 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.702100 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.702195 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":61,"thread_start_us":70,"threads_started":1}
I20260620 20:18:14.702917 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.695969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.0.0.1:36494)
0620 20:18:14.696275 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.696320 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.696423 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.698564 (+ 2141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.698567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.698585 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:14.698666 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.698672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.699967 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.699973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.701087 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.701094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.701823 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.702642 (+ 816us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.702662 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.702728 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.702817 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":754,"thread_start_us":65,"threads_started":1}
I20260620 20:18:14.703421 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.696813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.2.235.195:45271)
0620 20:18:14.697181 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.697195 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.697842 (+ 647us) server_negotiation.cc:407] Connection header received
0620 20:18:14.698014 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.698019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.698251 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.698659 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:14.698766 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.699621 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701093 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.701735 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.703112 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.703205 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.703238 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.703291 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20260620 20:18:14.703742 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.696097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.2.235.193:39785)
0620 20:18:14.698107 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.698117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.698138 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:18:14.698207 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.698212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.698417 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.698800 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:14.698939 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.699874 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701162 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.701687 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.701911 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.702038 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.703574 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.703644 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1907,"thread_start_us":106,"threads_started":1}
I20260620 20:18:14.705431 4995 tablet_service.cc:1467] Processing CreateTablet for tablet 05212888aa314b1bb9ff77e8b4ab4f33 (DEFAULT_TABLE table=test-table [id=8b59e7dbea7749f5b308221dd50bca28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:14.705904 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 05212888aa314b1bb9ff77e8b4ab4f33 (DEFAULT_TABLE table=test-table [id=8b59e7dbea7749f5b308221dd50bca28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:14.706430 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05212888aa314b1bb9ff77e8b4ab4f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.706583 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05212888aa314b1bb9ff77e8b4ab4f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.706377 4854 tablet_service.cc:1467] Processing CreateTablet for tablet 05212888aa314b1bb9ff77e8b4ab4f33 (DEFAULT_TABLE table=test-table [id=8b59e7dbea7749f5b308221dd50bca28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:14.707139 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05212888aa314b1bb9ff77e8b4ab4f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.712249 5224 tablet_bootstrap.cc:492] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Bootstrap starting.
I20260620 20:18:14.714174 5225 tablet_bootstrap.cc:492] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Bootstrap starting.
I20260620 20:18:14.714237 5226 tablet_bootstrap.cc:492] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Bootstrap starting.
I20260620 20:18:14.714910 5224 tablet_bootstrap.cc:654] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.715771 5224 log.cc:824] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:14.716113 5225 tablet_bootstrap.cc:654] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.716395 5226 tablet_bootstrap.cc:654] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.716822 5225 log.cc:824] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:14.717101 5226 log.cc:824] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:14.717780 5224 tablet_bootstrap.cc:492] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: No bootstrap required, opened a new log
I20260620 20:18:14.717950 5224 ts_tablet_manager.cc:1397] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:18:14.718400 5225 tablet_bootstrap.cc:492] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: No bootstrap required, opened a new log
I20260620 20:18:14.718569 5225 ts_tablet_manager.cc:1397] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:14.720852 5224 raft_consensus.cc:348] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.721117 5224 raft_consensus.cc:374] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.721174 5224 raft_consensus.cc:729] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a18f0089fdb459799a0ec663eb81796, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.721227 5225 raft_consensus.cc:348] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.721484 5225 raft_consensus.cc:374] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.721536 5225 raft_consensus.cc:729] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.721629 5224 consensus_queue.cc:260] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.721903 5225 consensus_queue.cc:260] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.722198 5201 heartbeater.cc:503] Master 127.2.235.254:40413 was elected leader, sending a full tablet report...
I20260620 20:18:14.722446 5061 heartbeater.cc:503] Master 127.2.235.254:40413 was elected leader, sending a full tablet report...
I20260620 20:18:14.722543 5224 ts_tablet_manager.cc:1428] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:18:14.722757 5225 ts_tablet_manager.cc:1428] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 20:18:14.723958 5226 tablet_bootstrap.cc:492] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: No bootstrap required, opened a new log
I20260620 20:18:14.724256 5226 ts_tablet_manager.cc:1397] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260620 20:18:14.727483 5226 raft_consensus.cc:348] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.727631 5226 raft_consensus.cc:374] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.727658 5226 raft_consensus.cc:729] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.727886 5226 consensus_queue.cc:260] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.728380 4921 heartbeater.cc:503] Master 127.2.235.254:40413 was elected leader, sending a full tablet report...
I20260620 20:18:14.728595 5226 ts_tablet_manager.cc:1428] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260620 20:18:14.738761 4922 tablet.cc:2369] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:14.741146 5232 raft_consensus.cc:482] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:14.741284 5232 raft_consensus.cc:504] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.742080 5232 leader_election.cc:288] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271), 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465)
I20260620 20:18:14.746255 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.742285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:45271 (local address 127.2.235.193:58547)
0620 20:18:14.742651 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.742661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.742766 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.742987 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.742990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.743004 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:14.743170 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.743175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.743994 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.743997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.745115 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.745125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.745887 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.745890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.745983 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.746006 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.746067 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.746114 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":132,"thread_start_us":102,"threads_started":1}
I20260620 20:18:14.746266 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.742358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:58547 (local address 127.2.235.195:45271)
0620 20:18:14.742450 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.742455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.742735 (+ 280us) server_negotiation.cc:407] Connection header received
0620 20:18:14.742860 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.742863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.742902 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.742968 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:14.743269 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.743879 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.745244 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.745777 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.745881 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.745921 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.746097 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.746172 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 20:18:14.746367 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.742371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:36289 (local address 127.2.235.194:42465)
0620 20:18:14.742456 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.742461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.742647 (+ 186us) server_negotiation.cc:407] Connection header received
0620 20:18:14.742783 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.742786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.742827 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.742896 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:14.743192 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.743729 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.745230 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.745794 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.745881 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.745921 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.746203 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.746269 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:18:14.746395 5233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.742278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:42465 (local address 127.2.235.193:36289)
0620 20:18:14.742570 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.742581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.742683 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.742914 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.742917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.742947 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:14.743092 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.743098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.743810 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.743813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.745085 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.745096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.745981 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.745983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.746087 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.746108 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.746171 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.746229 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":35,"thread_start_us":51,"threads_started":1}
I20260620 20:18:14.746902 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05212888aa314b1bb9ff77e8b4ab4f33" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796" is_pre_election: true
I20260620 20:18:14.746968 5015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05212888aa314b1bb9ff77e8b4ab4f33" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91db6e06089347b6a6fbc888a3693f1f" is_pre_election: true
I20260620 20:18:14.747131 5155 raft_consensus.cc:2454] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 0.
I20260620 20:18:14.747187 5015 raft_consensus.cc:2454] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 0.
I20260620 20:18:14.747509 4809 leader_election.cc:302] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a18f0089fdb459799a0ec663eb81796, 6fae93d0542a4e6db3acbd0c74016e81, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:14.747741 5232 raft_consensus.cc:2781] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:14.747836 5232 raft_consensus.cc:482] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:14.747885 5232 raft_consensus.cc:3037] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.749027 5232 raft_consensus.cc:504] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.749389 5232 leader_election.cc:288] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 election: Requested vote from peers 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271), 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465)
I20260620 20:18:14.749616 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05212888aa314b1bb9ff77e8b4ab4f33" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796"
I20260620 20:18:14.749647 5015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05212888aa314b1bb9ff77e8b4ab4f33" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91db6e06089347b6a6fbc888a3693f1f"
I20260620 20:18:14.749707 5155 raft_consensus.cc:3037] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.749758 5015 raft_consensus.cc:3037] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.750566 5155 raft_consensus.cc:2454] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 1.
I20260620 20:18:14.750581 5015 raft_consensus.cc:2454] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 1.
I20260620 20:18:14.750862 4809 leader_election.cc:302] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a18f0089fdb459799a0ec663eb81796, 6fae93d0542a4e6db3acbd0c74016e81; no voters:
I20260620 20:18:14.751019 5232 raft_consensus.cc:2781] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:14.751212 5232 raft_consensus.cc:686] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 LEADER]: Becoming Leader. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Running, Role: LEADER
I20260620 20:18:14.751472 5232 consensus_queue.cc:237] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } }
I20260620 20:18:14.753785 4718 catalog_manager.cc:5495] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193). New cstate: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } health_report { overall_health: UNKNOWN } } }
Jun 20 20:18:14 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986693, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:14.785741 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.778516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.2.235.254:40413)
0620 20:18:14.778650 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.778655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.778740 (+ 85us) server_negotiation.cc:407] Connection header received
0620 20:18:14.778816 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.778817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.778849 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.778899 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:14.779400 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.779834 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.780517 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.780695 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.780757 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.782758 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:14.782761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:14.782762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:14.782779 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:14.784114 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.784559 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.784562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.784566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.784641 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:14.784903 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:14.784906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:14.784908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:14.785067 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:14.785168 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.785474 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.785602 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":70}
I20260620 20:18:14.785741 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.778442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40413 (local address 127.0.0.1:39868)
0620 20:18:14.778665 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.778675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.778752 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.778933 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.778935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.779238 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:14.779313 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.779318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.779917 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.779919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.780433 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.780439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.780775 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.780777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.780831 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.781373 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:14.781383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:14.782636 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:14.784234 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.784244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.784250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.784467 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.784729 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:14.784733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:14.784736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:14.784814 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:14.785167 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:14.785173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:14.785302 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.785426 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.785562 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260620 20:18:14.788960 4718 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:39868:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:18:14.789606 4718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:14.793699 4854 tablet_service.cc:1467] Processing CreateTablet for tablet 1328a61287d84d298411cf54fbd1177d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b41e70886e2a4b7b8e5153df191b36e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:14.793977 4995 tablet_service.cc:1467] Processing CreateTablet for tablet 1328a61287d84d298411cf54fbd1177d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b41e70886e2a4b7b8e5153df191b36e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:14.794399 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1328a61287d84d298411cf54fbd1177d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.794589 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1328a61287d84d298411cf54fbd1177d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.794273 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 1328a61287d84d298411cf54fbd1177d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b41e70886e2a4b7b8e5153df191b36e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:14.794803 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1328a61287d84d298411cf54fbd1177d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.796727 5225 tablet_bootstrap.cc:492] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: Bootstrap starting.
I20260620 20:18:14.797053 5226 tablet_bootstrap.cc:492] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: Bootstrap starting.
I20260620 20:18:14.797441 5224 tablet_bootstrap.cc:492] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: Bootstrap starting.
I20260620 20:18:14.797744 5225 tablet_bootstrap.cc:654] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.798013 5226 tablet_bootstrap.cc:654] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.798130 5224 tablet_bootstrap.cc:654] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.798972 5225 tablet_bootstrap.cc:492] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: No bootstrap required, opened a new log
I20260620 20:18:14.799057 5225 ts_tablet_manager.cc:1397] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:14.799223 5224 tablet_bootstrap.cc:492] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: No bootstrap required, opened a new log
I20260620 20:18:14.799296 5224 ts_tablet_manager.cc:1397] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:14.799407 5225 raft_consensus.cc:348] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.799463 5226 tablet_bootstrap.cc:492] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: No bootstrap required, opened a new log
I20260620 20:18:14.799472 5225 raft_consensus.cc:374] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.799501 5225 raft_consensus.cc:729] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.799532 5226 ts_tablet_manager.cc:1397] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:14.799612 5225 consensus_queue.cc:260] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.799710 5224 raft_consensus.cc:348] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.799793 5224 raft_consensus.cc:374] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.799819 5224 raft_consensus.cc:729] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a18f0089fdb459799a0ec663eb81796, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.799860 5225 ts_tablet_manager.cc:1428] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:14.799825 5231 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.799938 5231 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:14.799894 5226 raft_consensus.cc:348] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.799919 5224 consensus_queue.cc:260] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.799973 5226 raft_consensus.cc:374] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.800000 5226 raft_consensus.cc:729] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.800103 5226 consensus_queue.cc:260] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.800137 5230 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.800256 5230 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:14.800341 5226 ts_tablet_manager.cc:1428] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:14.800590 5232 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.800688 5232 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:14.800146 5224 ts_tablet_manager.cc:1428] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:14.855952 5232 raft_consensus.cc:482] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:14.856037 5232 raft_consensus.cc:504] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.856411 5232 leader_election.cc:288] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465), 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:14.856755 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1328a61287d84d298411cf54fbd1177d" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796" is_pre_election: true
I20260620 20:18:14.856809 5015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1328a61287d84d298411cf54fbd1177d" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91db6e06089347b6a6fbc888a3693f1f" is_pre_election: true
I20260620 20:18:14.856904 5155 raft_consensus.cc:2454] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 0.
I20260620 20:18:14.856926 5015 raft_consensus.cc:2454] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 0.
I20260620 20:18:14.857188 4808 leader_election.cc:302] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fae93d0542a4e6db3acbd0c74016e81, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:14.857376 5232 raft_consensus.cc:2781] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:14.857434 5232 raft_consensus.cc:482] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:14.857460 5232 raft_consensus.cc:3037] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.858276 5232 raft_consensus.cc:504] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.858635 5232 leader_election.cc:288] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 election: Requested vote from peers 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465), 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:14.858875 5015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1328a61287d84d298411cf54fbd1177d" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91db6e06089347b6a6fbc888a3693f1f"
I20260620 20:18:14.858987 5015 raft_consensus.cc:3037] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.858976 5155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1328a61287d84d298411cf54fbd1177d" candidate_uuid: "6fae93d0542a4e6db3acbd0c74016e81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796"
I20260620 20:18:14.859066 5155 raft_consensus.cc:3037] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:14.860060 5155 raft_consensus.cc:2454] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 1.
I20260620 20:18:14.860060 5015 raft_consensus.cc:2454] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fae93d0542a4e6db3acbd0c74016e81 in term 1.
I20260620 20:18:14.860361 4808 leader_election.cc:302] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fae93d0542a4e6db3acbd0c74016e81, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:14.860512 5232 raft_consensus.cc:2781] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:14.860594 5232 raft_consensus.cc:686] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 LEADER]: Becoming Leader. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Running, Role: LEADER
I20260620 20:18:14.860704 5232 consensus_queue.cc:237] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.861333 5235 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: TxnStatusTablet state changed. Reason: New leader 6fae93d0542a4e6db3acbd0c74016e81. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.861443 5235 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:14.861759 5247 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:18:14.861979 5247 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:18:14.862246 4716 catalog_manager.cc:5495] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193). New cstate: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:14.883754 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.879928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.0.0.1:36498)
0620 20:18:14.880083 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.880099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.880195 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.880484 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.880487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.880492 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:14.880577 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.880581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.881577 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.881580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.882164 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.882171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.882734 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.882736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.882802 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.882842 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:14.883522 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:14.883535 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.883592 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.883656 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:18:14.883775 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.879986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.2.235.193:39785)
0620 20:18:14.880091 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.880096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.880147 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:18:14.880315 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.880318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.880373 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.880471 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:14.880668 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.881470 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.882270 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.882629 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.882729 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.882975 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:14.883424 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:14.883531 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.883603 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.883660 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:14.888420 5155 raft_consensus.cc:1261] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Refusing update from remote peer 6fae93d0542a4e6db3acbd0c74016e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:14.888469 5015 raft_consensus.cc:1261] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Refusing update from remote peer 6fae93d0542a4e6db3acbd0c74016e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:14.889058 5235 consensus_queue.cc:1035] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:14.889238 5232 consensus_queue.cc:1035] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:14.892828 5230 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: TxnStatusTablet state changed. Reason: New leader 6fae93d0542a4e6db3acbd0c74016e81. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.892936 5230 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:14.893127 5231 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: TxnStatusTablet state changed. Reason: New leader 6fae93d0542a4e6db3acbd0c74016e81. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.893224 5231 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 20:18:14.894285 5202 tablet.cc:2369] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:14.895753 5232 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.895848 5232 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:14.896052 5230 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.896140 5230 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 4a18f0089fdb459799a0ec663eb81796: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:14.896970 5250 mvcc.cc:205] Tried to move back new op lower bound from 7299017502256861184 to 7299017502150746112. Current Snapshot: MvccSnapshot[applied={T|T < 7299017502256861184}]
I20260620 20:18:14.898455 5252 mvcc.cc:205] Tried to move back new op lower bound from 7299017502256861184 to 7299017502150746112. Current Snapshot: MvccSnapshot[applied={T|T < 7299017502256861184}]
I20260620 20:18:14.898633 5235 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.898720 5235 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 6fae93d0542a4e6db3acbd0c74016e81: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:14.899451 5251 mvcc.cc:205] Tried to move back new op lower bound from 7299017502256861184 to 7299017502150746112. Current Snapshot: MvccSnapshot[applied={T|T < 7299017502256861184}]
I20260620 20:18:14.900707 5231 tablet_replica.cc:439] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fae93d0542a4e6db3acbd0c74016e81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } } }
I20260620 20:18:14.900810 5231 tablet_replica.cc:442] T 1328a61287d84d298411cf54fbd1177d P 91db6e06089347b6a6fbc888a3693f1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:14.909839 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.905992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.2.235.193:39785)
0620 20:18:14.906335 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.906339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.906350 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:14.906384 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.906387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.906424 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.906490 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:14.906728 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.907414 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.908256 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.908662 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.908796 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.909164 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:14.909227 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:14.909335 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.909647 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.909700 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260620 20:18:14.922231 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.918307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:56613 (local address 127.2.235.193:39785)
0620 20:18:14.918461 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:14.918465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:14.918734 (+ 269us) server_negotiation.cc:407] Connection header received
0620 20:18:14.918908 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:14.918912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:14.918979 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:14.919063 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:14.919422 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.919875 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.921107 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.921404 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.921486 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.921836 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:14.921889 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:14.921943 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:14.922078 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:18:14.922112 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260620 20:18:14.922397 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:14.918170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.2.235.193:56613)
0620 20:18:14.918641 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:14.918655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:14.918757 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:14.919086 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:14.919089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:14.919096 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:14.919303 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.919311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.919977 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.919981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.920952 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:14.920963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:14.921537 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:14.921542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:14.921637 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:14.921686 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:14.921969 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:14.921975 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:18:14.922051 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:18:14.922125 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260620 20:18:14.928118 5155 raft_consensus.cc:1261] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Refusing update from remote peer 6fae93d0542a4e6db3acbd0c74016e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:14.928457 5015 raft_consensus.cc:1261] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Refusing update from remote peer 6fae93d0542a4e6db3acbd0c74016e81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:14.928508 5232 consensus_queue.cc:1035] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:14.928784 5232 consensus_queue.cc:1035] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:14.964694 4717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:14.964857 4717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:14.967146 4717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Sending DeleteTablet for 3 replicas of tablet 05212888aa314b1bb9ff77e8b4ab4f33
I20260620 20:18:14.967727 4854 tablet_service.cc:1514] Processing DeleteTablet for tablet 05212888aa314b1bb9ff77e8b4ab4f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:14 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:36494
I20260620 20:18:14.967851 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 05212888aa314b1bb9ff77e8b4ab4f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:14 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 20:18:14.967931 4995 tablet_service.cc:1514] Processing DeleteTablet for tablet 05212888aa314b1bb9ff77e8b4ab4f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:14 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:59568
I20260620 20:18:14.968225 5267 tablet_replica.cc:330] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: stopping tablet replica
I20260620 20:18:14.968269 5268 tablet_replica.cc:330] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: stopping tablet replica
I20260620 20:18:14.968510 5267 raft_consensus.cc:2229] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:14.968724 5268 raft_consensus.cc:2229] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:14.968819 5267 raft_consensus.cc:2258] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:14.968979 5268 raft_consensus.cc:2258] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:14.969002 4717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:14.969378 4717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:14.969961 5267 ts_tablet_manager.cc:1905] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:14.972728 5267 ts_tablet_manager.cc:1918] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:14.972824 5267 log.cc:1196] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/05212888aa314b1bb9ff77e8b4ab4f33
I20260620 20:18:14.973037 5268 ts_tablet_manager.cc:1905] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:14.973136 5267 ts_tablet_manager.cc:1939] T 05212888aa314b1bb9ff77e8b4ab4f33 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting consensus metadata
I20260620 20:18:14.974392 4995 tablet_service.cc:1467] Processing CreateTablet for tablet 3b740603a3ee4688897d6dc32e1fbdd7 (DEFAULT_TABLE table=test-table [id=1e240ee54c5a464883b0c76d2525e3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:14.974455 4854 tablet_service.cc:1467] Processing CreateTablet for tablet 3b740603a3ee4688897d6dc32e1fbdd7 (DEFAULT_TABLE table=test-table [id=1e240ee54c5a464883b0c76d2525e3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:14.974647 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b740603a3ee4688897d6dc32e1fbdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.974617 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 3b740603a3ee4688897d6dc32e1fbdd7 (DEFAULT_TABLE table=test-table [id=1e240ee54c5a464883b0c76d2525e3ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:14.974671 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b740603a3ee4688897d6dc32e1fbdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.974830 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b740603a3ee4688897d6dc32e1fbdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:14.976617 5226 tablet_bootstrap.cc:492] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Bootstrap starting.
I20260620 20:18:14.976971 5225 tablet_bootstrap.cc:492] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Bootstrap starting.
I20260620 20:18:14.977161 5224 tablet_bootstrap.cc:492] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Bootstrap starting.
I20260620 20:18:14.977409 5269 tablet_replica.cc:330] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: stopping tablet replica
I20260620 20:18:14.977540 5226 tablet_bootstrap.cc:654] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.977658 5269 raft_consensus.cc:2229] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:14.977839 5225 tablet_bootstrap.cc:654] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.977846 5269 raft_consensus.cc:2258] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:14.978060 5224 tablet_bootstrap.cc:654] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:14.978820 4705 catalog_manager.cc:4841] TS 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785): tablet 05212888aa314b1bb9ff77e8b4ab4f33 (table test-table [id=8b59e7dbea7749f5b308221dd50bca28]) successfully deleted
I20260620 20:18:14.979148 5225 tablet_bootstrap.cc:492] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: No bootstrap required, opened a new log
I20260620 20:18:14.979231 5225 ts_tablet_manager.cc:1397] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:14.979516 5226 tablet_bootstrap.cc:492] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: No bootstrap required, opened a new log
I20260620 20:18:14.979610 5226 ts_tablet_manager.cc:1397] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:18:14.979650 5225 raft_consensus.cc:348] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.979784 5225 raft_consensus.cc:374] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.979825 5225 raft_consensus.cc:729] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.979941 5225 consensus_queue.cc:260] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.980046 5226 raft_consensus.cc:348] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.980146 5226 raft_consensus.cc:374] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.980183 5226 raft_consensus.cc:729] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.980212 5225 ts_tablet_manager.cc:1428] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:14.980296 5226 consensus_queue.cc:260] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.980556 5226 ts_tablet_manager.cc:1428] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:14.982662 5269 ts_tablet_manager.cc:1905] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:14.985954 5269 ts_tablet_manager.cc:1918] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:14.986097 5269 log.cc:1196] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/05212888aa314b1bb9ff77e8b4ab4f33
I20260620 20:18:14.986351 5269 ts_tablet_manager.cc:1939] T 05212888aa314b1bb9ff77e8b4ab4f33 P 4a18f0089fdb459799a0ec663eb81796: Deleting consensus metadata
I20260620 20:18:14.986724 5268 ts_tablet_manager.cc:1918] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:14.986812 5268 log.cc:1196] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/05212888aa314b1bb9ff77e8b4ab4f33
I20260620 20:18:14.987100 5268 ts_tablet_manager.cc:1939] T 05212888aa314b1bb9ff77e8b4ab4f33 P 91db6e06089347b6a6fbc888a3693f1f: Deleting consensus metadata
I20260620 20:18:14.987286 4705 catalog_manager.cc:4841] TS 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271): tablet 05212888aa314b1bb9ff77e8b4ab4f33 (table test-table [id=8b59e7dbea7749f5b308221dd50bca28]) successfully deleted
I20260620 20:18:14.988052 5224 tablet_bootstrap.cc:492] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: No bootstrap required, opened a new log
I20260620 20:18:14.988111 4704 catalog_manager.cc:4841] TS 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465): tablet 05212888aa314b1bb9ff77e8b4ab4f33 (table test-table [id=8b59e7dbea7749f5b308221dd50bca28]) successfully deleted
I20260620 20:18:14.988155 5224 ts_tablet_manager.cc:1397] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 20:18:14.988665 5224 raft_consensus.cc:348] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.988770 5224 raft_consensus.cc:374] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:14.988826 5224 raft_consensus.cc:729] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a18f0089fdb459799a0ec663eb81796, State: Initialized, Role: FOLLOWER
I20260620 20:18:14.988950 5224 consensus_queue.cc:260] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:14.989240 5224 ts_tablet_manager.cc:1428] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:15.037982 5230 raft_consensus.cc:482] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:15.038107 5230 raft_consensus.cc:504] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:15.038990 5230 leader_election.cc:288] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785), 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465)
I20260620 20:18:15.043116 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.039190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:42465 (local address 127.2.235.195:52127)
0620 20:18:15.039303 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:15.039317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:15.039393 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:15.039715 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:15.039718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:15.039737 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:15.039852 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.039862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.040707 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.040710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.041936 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.041947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.042734 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.042737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.042838 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.042861 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:18:15.042923 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 20:18:15.042989 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 20:18:15.043201 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.039360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:52127 (local address 127.2.235.194:42465)
0620 20:18:15.039490 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:15.039494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:15.039506 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:15.039551 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:15.039554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:15.039600 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:15.039667 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:15.039974 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.040569 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.042088 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.042628 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.042735 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.042783 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:15.042985 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 20:18:15.043063 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:18:15.043277 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.039348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:53567 (local address 127.2.235.193:39785)
0620 20:18:15.039457 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:15.039462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:15.039845 (+ 383us) server_negotiation.cc:407] Connection header received
0620 20:18:15.040028 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:15.040032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:15.040079 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:15.040127 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:15.040508 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.041060 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.042221 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.042748 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.042863 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.042913 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:15.043076 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 20:18:15.043151 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:18:15.043606 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.039192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.2.235.195:53567)
0620 20:18:15.039764 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:15.039778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:15.039912 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:15.040256 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:15.040259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:15.040274 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:15.040391 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.040397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.041168 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.041172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.042068 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.042079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.042858 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.042861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.042949 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.042965 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 20:18:15.043401 (+ 436us) client_negotiation.cc:240] Negotiation successful
0620 20:18:15.043488 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":281,"thread_start_us":79,"threads_started":1}
I20260620 20:18:15.043673 5014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b740603a3ee4688897d6dc32e1fbdd7" candidate_uuid: "4a18f0089fdb459799a0ec663eb81796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91db6e06089347b6a6fbc888a3693f1f" is_pre_election: true
I20260620 20:18:15.043814 5014 raft_consensus.cc:2454] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a18f0089fdb459799a0ec663eb81796 in term 0.
I20260620 20:18:15.044149 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b740603a3ee4688897d6dc32e1fbdd7" candidate_uuid: "4a18f0089fdb459799a0ec663eb81796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fae93d0542a4e6db3acbd0c74016e81" is_pre_election: true
I20260620 20:18:15.044175 5088 leader_election.cc:302] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a18f0089fdb459799a0ec663eb81796, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:15.044354 4875 raft_consensus.cc:2454] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a18f0089fdb459799a0ec663eb81796 in term 0.
I20260620 20:18:15.044412 5230 raft_consensus.cc:2781] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:15.044480 5230 raft_consensus.cc:482] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:15.044528 5230 raft_consensus.cc:3037] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:15.045751 5230 raft_consensus.cc:504] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:15.046073 5230 leader_election.cc:288] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [CANDIDATE]: Term 1 election: Requested vote from peers 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785), 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465)
I20260620 20:18:15.046320 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b740603a3ee4688897d6dc32e1fbdd7" candidate_uuid: "4a18f0089fdb459799a0ec663eb81796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fae93d0542a4e6db3acbd0c74016e81"
I20260620 20:18:15.046376 5014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b740603a3ee4688897d6dc32e1fbdd7" candidate_uuid: "4a18f0089fdb459799a0ec663eb81796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91db6e06089347b6a6fbc888a3693f1f"
I20260620 20:18:15.046464 4875 raft_consensus.cc:3037] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:15.046532 5014 raft_consensus.cc:3037] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:15.047268 4875 raft_consensus.cc:2454] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a18f0089fdb459799a0ec663eb81796 in term 1.
I20260620 20:18:15.047578 5089 leader_election.cc:302] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a18f0089fdb459799a0ec663eb81796, 6fae93d0542a4e6db3acbd0c74016e81; no voters:
I20260620 20:18:15.047678 5014 raft_consensus.cc:2454] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a18f0089fdb459799a0ec663eb81796 in term 1.
I20260620 20:18:15.047767 5230 raft_consensus.cc:2781] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:15.047952 5230 raft_consensus.cc:686] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 1 LEADER]: Becoming Leader. State: Replica: 4a18f0089fdb459799a0ec663eb81796, State: Running, Role: LEADER
I20260620 20:18:15.048172 5230 consensus_queue.cc:237] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:15.050308 4718 catalog_manager.cc:5495] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "4a18f0089fdb459799a0ec663eb81796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } health_report { overall_health: HEALTHY } } }
I20260620 20:18:15.067706 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.063964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.2.235.195:45271)
0620 20:18:15.064096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:15.064101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:15.064219 (+ 118us) server_negotiation.cc:407] Connection header received
0620 20:18:15.064339 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:15.064343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:15.064402 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:15.064504 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:15.064709 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.065457 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.066182 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.066479 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.066588 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.066794 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:15.067268 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:15.067401 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:15.067538 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:18:15.067587 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:15.077373 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.074029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.2.235.195:43803)
0620 20:18:15.074193 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:15.074209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:15.074284 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:15.074495 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:15.074496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:15.074501 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:15.074577 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.074581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.075468 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.075470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.075975 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.075982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.076668 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.076670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.076731 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.076765 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:15.077102 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:15.077106 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:18:15.077162 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 20:18:15.077242 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:18:15.077392 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.074148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:43803 (local address 127.2.235.193:39785)
0620 20:18:15.074252 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:15.074256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:15.074272 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:15.074353 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:15.074357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:15.074404 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:15.074489 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:15.074670 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.075378 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.076079 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.076419 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.076541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.076939 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:15.077012 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:15.077083 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:15.077206 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 20:18:15.077272 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:15.082945 5014 raft_consensus.cc:1261] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Refusing update from remote peer 4a18f0089fdb459799a0ec663eb81796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:15.083138 4875 raft_consensus.cc:1261] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Refusing update from remote peer 4a18f0089fdb459799a0ec663eb81796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:15.083323 5230 consensus_queue.cc:1035] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:15.083590 5274 consensus_queue.cc:1035] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:15.103299 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.099812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:45271 (local address 127.2.235.193:48075)
0620 20:18:15.100068 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:15.100083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:15.100176 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:15.100527 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:15.100529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:15.100536 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:15.100608 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.100612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.101584 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.101588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.102281 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:15.102289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.102792 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.102795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:15.102849 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.102871 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:15.103114 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:15.103118 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:18:15.103162 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 20:18:15.103217 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 20:18:15.103327 5222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:15.100039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:48075 (local address 127.2.235.195:45271)
0620 20:18:15.100318 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:15.100323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:15.100341 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:18:15.100380 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:15.100383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:15.100427 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:15.100504 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:15.100726 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.101480 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.102391 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:15.102695 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:15.102793 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:15.102970 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:15.103038 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:15.103108 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:15.103174 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 20:18:15.103218 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
W20260620 20:18:15.103912 5252 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:18:16.666625 4718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:16.666760 4718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:16.668838 4718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Sending DeleteTablet for 3 replicas of tablet 3b740603a3ee4688897d6dc32e1fbdd7
I20260620 20:18:16.669320 4854 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b740603a3ee4688897d6dc32e1fbdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:16 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:36494
I20260620 20:18:16.669389 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b740603a3ee4688897d6dc32e1fbdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:16 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 20:18:16.669411 4995 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b740603a3ee4688897d6dc32e1fbdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:16 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:59568
I20260620 20:18:16.669703 5305 tablet_replica.cc:330] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: stopping tablet replica
I20260620 20:18:16.669700 5307 tablet_replica.cc:330] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: stopping tablet replica
I20260620 20:18:16.669835 5305 raft_consensus.cc:2229] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:16.669941 5305 raft_consensus.cc:2258] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:16.669965 5307 raft_consensus.cc:2229] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:16.670094 5307 raft_consensus.cc:2258] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:16.670523 5305 ts_tablet_manager.cc:1905] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:16.670709 5307 ts_tablet_manager.cc:1905] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:16.672470 5306 tablet_replica.cc:330] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: stopping tablet replica
I20260620 20:18:16.672606 5306 raft_consensus.cc:2229] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:16.672864 5306 raft_consensus.cc:2258] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:16.673296 5305 ts_tablet_manager.cc:1918] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:18:16.673382 5305 log.cc:1196] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/3b740603a3ee4688897d6dc32e1fbdd7
I20260620 20:18:16.673401 5307 ts_tablet_manager.cc:1918] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:18:16.673456 5306 ts_tablet_manager.cc:1905] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:16.673472 5307 log.cc:1196] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/3b740603a3ee4688897d6dc32e1fbdd7
I20260620 20:18:16.673650 5305 ts_tablet_manager.cc:1939] T 3b740603a3ee4688897d6dc32e1fbdd7 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting consensus metadata
I20260620 20:18:16.673758 5307 ts_tablet_manager.cc:1939] T 3b740603a3ee4688897d6dc32e1fbdd7 P 91db6e06089347b6a6fbc888a3693f1f: Deleting consensus metadata
I20260620 20:18:16.674324 4704 catalog_manager.cc:4841] TS 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465): tablet 3b740603a3ee4688897d6dc32e1fbdd7 (table test-table [id=1e240ee54c5a464883b0c76d2525e3ce]) successfully deleted
I20260620 20:18:16.674329 4705 catalog_manager.cc:4841] TS 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785): tablet 3b740603a3ee4688897d6dc32e1fbdd7 (table test-table [id=1e240ee54c5a464883b0c76d2525e3ce]) successfully deleted
I20260620 20:18:16.675846 5306 ts_tablet_manager.cc:1918] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:18:16.675904 5306 log.cc:1196] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/3b740603a3ee4688897d6dc32e1fbdd7
I20260620 20:18:16.676172 5306 ts_tablet_manager.cc:1939] T 3b740603a3ee4688897d6dc32e1fbdd7 P 4a18f0089fdb459799a0ec663eb81796: Deleting consensus metadata
I20260620 20:18:16.676671 4705 catalog_manager.cc:4841] TS 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271): tablet 3b740603a3ee4688897d6dc32e1fbdd7 (table test-table [id=1e240ee54c5a464883b0c76d2525e3ce]) successfully deleted
Jun 20 20:18:16 dist-test-slave-8z53 krb5kdc[4673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986696, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:18:16 dist-test-slave-8z53 krb5kdc[4673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986696, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:16.695039 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.687133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.2.235.254:40413)
0620 20:18:16.687357 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:16.687367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:16.687434 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:18:16.687584 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:16.687587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:16.687630 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:16.687728 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:16.688331 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.688852 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.689654 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.689867 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.689966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.692155 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:16.692165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:16.692167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:16.692188 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:16.693488 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:16.693853 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:16.693858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:16.693861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:16.693932 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:16.694230 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:16.694234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:16.694236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:16.694402 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:16.694547 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:16.694752 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 20:18:16.694871 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260620 20:18:16.697281 4718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:16.697670 4718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:16.701583 4854 tablet_service.cc:1467] Processing CreateTablet for tablet 483a43802c914326a77fab974c724186 (DEFAULT_TABLE table=test-table [id=673f54780c6d42b0b38d2325080fedeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:16.701620 4995 tablet_service.cc:1467] Processing CreateTablet for tablet 483a43802c914326a77fab974c724186 (DEFAULT_TABLE table=test-table [id=673f54780c6d42b0b38d2325080fedeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:16.701879 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483a43802c914326a77fab974c724186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:16.701910 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483a43802c914326a77fab974c724186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:16.704129 5316 tablet_bootstrap.cc:492] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Bootstrap starting.
I20260620 20:18:16.704671 5135 tablet_service.cc:1467] Processing CreateTablet for tablet 483a43802c914326a77fab974c724186 (DEFAULT_TABLE table=test-table [id=673f54780c6d42b0b38d2325080fedeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:16.704936 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483a43802c914326a77fab974c724186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:16.705199 5316 tablet_bootstrap.cc:654] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:16.705693 5317 tablet_bootstrap.cc:492] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Bootstrap starting.
I20260620 20:18:16.706715 5316 tablet_bootstrap.cc:492] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: No bootstrap required, opened a new log
I20260620 20:18:16.706756 5317 tablet_bootstrap.cc:654] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:16.706768 5316 ts_tablet_manager.cc:1397] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:16.707026 5316 raft_consensus.cc:348] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.707116 5316 raft_consensus.cc:374] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:16.707144 5316 raft_consensus.cc:729] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Initialized, Role: FOLLOWER
I20260620 20:18:16.707245 5319 tablet_bootstrap.cc:492] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Bootstrap starting.
I20260620 20:18:16.707271 5316 consensus_queue.cc:260] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.707496 5316 ts_tablet_manager.cc:1428] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:16.707949 5319 tablet_bootstrap.cc:654] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:16.708184 5317 tablet_bootstrap.cc:492] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: No bootstrap required, opened a new log
I20260620 20:18:16.708276 5317 ts_tablet_manager.cc:1397] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:16.708658 5317 raft_consensus.cc:348] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.708755 5317 raft_consensus.cc:374] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:16.708788 5317 raft_consensus.cc:729] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Initialized, Role: FOLLOWER
I20260620 20:18:16.709017 5317 consensus_queue.cc:260] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.709110 5322 raft_consensus.cc:482] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:16.709170 5322 raft_consensus.cc:504] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.709744 5317 ts_tablet_manager.cc:1428] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:16.709841 5319 tablet_bootstrap.cc:492] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: No bootstrap required, opened a new log
I20260620 20:18:16.709949 5319 ts_tablet_manager.cc:1397] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:16.710332 5319 raft_consensus.cc:348] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.710430 5319 raft_consensus.cc:374] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:16.710466 5319 raft_consensus.cc:729] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a18f0089fdb459799a0ec663eb81796, State: Initialized, Role: FOLLOWER
I20260620 20:18:16.710604 5319 consensus_queue.cc:260] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.710866 5319 ts_tablet_manager.cc:1428] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:16.711094 5322 leader_election.cc:288] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785), 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:16.714830 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.710359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.2.235.194:49545)
0620 20:18:16.711283 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:16.711298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:16.711394 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:16.711750 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:16.711753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:16.711766 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:16.711874 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.711881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.712833 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.712835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.713738 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.713749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.714416 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.714420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.714521 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.714556 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:18:16.714623 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:18:16.714689 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":78,"threads_started":1}
I20260620 20:18:16.715041 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.710652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:49545 (local address 127.2.235.193:39785)
0620 20:18:16.711020 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:16.711040 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:16.711465 (+ 425us) server_negotiation.cc:407] Connection header received
0620 20:18:16.711505 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:16.711507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:16.711558 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:16.711631 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:16.711977 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.712707 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.713846 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.714294 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.714416 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.714525 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:16.714634 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 20:18:16.714700 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260620 20:18:16.715358 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.711252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:45271 (local address 127.2.235.194:60261)
0620 20:18:16.711770 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:16.711782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:16.711845 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:16.712248 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:16.712251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:16.712265 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:16.712350 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.712355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.713344 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.713347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.714111 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.714119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.714917 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.714920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.715015 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.715046 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:18:16.715137 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:18:16.715210 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260620 20:18:16.715373 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a43802c914326a77fab974c724186" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fae93d0542a4e6db3acbd0c74016e81" is_pre_election: true
I20260620 20:18:16.715490 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.711536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:60261 (local address 127.2.235.195:45271)
0620 20:18:16.711996 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:16.712004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:16.712014 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:18:16.712057 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:16.712060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:16.712097 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:16.712162 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:16.712455 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.713194 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.714269 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.714803 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.714923 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.714967 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:16.715269 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 20:18:16.715355 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":74,"threads_started":1}
I20260620 20:18:16.715515 4875 raft_consensus.cc:2454] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 0.
I20260620 20:18:16.715806 4949 leader_election.cc:302] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fae93d0542a4e6db3acbd0c74016e81, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:16.715869 5154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a43802c914326a77fab974c724186" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796" is_pre_election: true
I20260620 20:18:16.715972 5154 raft_consensus.cc:2454] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 0.
I20260620 20:18:16.716019 5322 raft_consensus.cc:2781] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:16.716078 5322 raft_consensus.cc:482] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:16.716123 5322 raft_consensus.cc:3037] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:16.717104 5322 raft_consensus.cc:504] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.717413 5322 leader_election.cc:288] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 election: Requested vote from peers 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785), 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:16.717689 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a43802c914326a77fab974c724186" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fae93d0542a4e6db3acbd0c74016e81"
I20260620 20:18:16.717706 5154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a43802c914326a77fab974c724186" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796"
I20260620 20:18:16.717787 4875 raft_consensus.cc:3037] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:16.717818 5154 raft_consensus.cc:3037] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:16.718489 4875 raft_consensus.cc:2454] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 1.
I20260620 20:18:16.718621 5154 raft_consensus.cc:2454] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 1.
I20260620 20:18:16.718729 4949 leader_election.cc:302] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fae93d0542a4e6db3acbd0c74016e81, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:16.719022 5322 raft_consensus.cc:2781] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:16.719100 5322 raft_consensus.cc:686] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 1 LEADER]: Becoming Leader. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Running, Role: LEADER
I20260620 20:18:16.719290 5322 consensus_queue.cc:237] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.721230 4718 catalog_manager.cc:5495] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194). New cstate: current_term: 1 leader_uuid: "91db6e06089347b6a6fbc888a3693f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:16.737746 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.734255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.2.235.194:42465)
0620 20:18:16.734465 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:16.734470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:16.734483 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:16.734543 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:16.734545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:16.734587 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:16.734679 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:16.734874 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.735375 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.736093 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.736282 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.736359 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.736746 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:16.737344 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:16.737443 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:16.737584 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:18:16.737624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260620 20:18:16.745885 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.743215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:34413 (local address 127.2.235.193:39785)
0620 20:18:16.743304 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:16.743307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:16.743425 (+ 118us) server_negotiation.cc:407] Connection header received
0620 20:18:16.743501 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:16.743503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:16.743535 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:16.743586 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:16.743776 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.744171 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.744999 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.745207 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.745373 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.745626 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:16.745674 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:16.745718 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:16.745782 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:18:16.745813 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:18:16.745932 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.743065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.2.235.194:34413)
0620 20:18:16.743365 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:16.743374 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:16.743441 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:16.743605 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:16.743606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:16.743611 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:16.743718 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.743722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.744319 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.744322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.744910 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.744917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.745400 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.745403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.745501 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.745533 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:16.745733 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:16.745737 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:18:16.745775 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 20:18:16.745825 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":80,"threads_started":1}
I20260620 20:18:16.750438 4875 raft_consensus.cc:1261] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Refusing update from remote peer 91db6e06089347b6a6fbc888a3693f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:16.750478 5154 raft_consensus.cc:1261] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Refusing update from remote peer 91db6e06089347b6a6fbc888a3693f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:16.750847 5323 consensus_queue.cc:1035] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:16.751034 5322 consensus_queue.cc:1035] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:16.765601 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.762654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:48261 (local address 127.2.235.194:42465)
0620 20:18:16.762800 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:16.762804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:16.763030 (+ 226us) server_negotiation.cc:407] Connection header received
0620 20:18:16.763097 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:16.763099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:16.763130 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:16.763175 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:16.763315 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.763708 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.764522 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.764782 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.764891 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.765130 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:16.765198 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:16.765304 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:16.765445 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:18:16.765496 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:18:16.765638 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:16.762623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:42465 (local address 127.2.235.193:48261)
0620 20:18:16.762942 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:16.762953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:16.763034 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:16.763177 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:16.763179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:16.763183 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:16.763244 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.763247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.763783 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.763785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.764412 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:16.764423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:16.764901 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:16.764905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:16.764997 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:16.765019 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:16.765295 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:16.765302 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:18:16.765423 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:18:16.765501 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260620 20:18:16.781967 4718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:16.782068 4718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:16.783722 4718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Sending DeleteTablet for 3 replicas of tablet 483a43802c914326a77fab974c724186
I20260620 20:18:16.784240 4854 tablet_service.cc:1514] Processing DeleteTablet for tablet 483a43802c914326a77fab974c724186 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:16 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:36494
I20260620 20:18:16.784248 4995 tablet_service.cc:1514] Processing DeleteTablet for tablet 483a43802c914326a77fab974c724186 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:16 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:59568
I20260620 20:18:16.784418 5307 tablet_replica.cc:330] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: stopping tablet replica
I20260620 20:18:16.784420 5305 tablet_replica.cc:330] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: stopping tablet replica
I20260620 20:18:16.784526 5305 raft_consensus.cc:2229] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:16.784521 5307 raft_consensus.cc:2229] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:16.784623 5305 raft_consensus.cc:2258] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:16.784771 5307 raft_consensus.cc:2258] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:16.785372 5305 ts_tablet_manager.cc:1905] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:16.785585 4718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:18:16.785666 5307 ts_tablet_manager.cc:1905] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:18:16.785907 4718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:16.787809 5305 ts_tablet_manager.cc:1918] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:16.787809 5307 ts_tablet_manager.cc:1918] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:16.787875 5305 log.cc:1196] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/483a43802c914326a77fab974c724186
I20260620 20:18:16.787875 5307 log.cc:1196] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/483a43802c914326a77fab974c724186
I20260620 20:18:16.788113 5307 ts_tablet_manager.cc:1939] T 483a43802c914326a77fab974c724186 P 91db6e06089347b6a6fbc888a3693f1f: Deleting consensus metadata
I20260620 20:18:16.788115 5305 ts_tablet_manager.cc:1939] T 483a43802c914326a77fab974c724186 P 6fae93d0542a4e6db3acbd0c74016e81: Deleting consensus metadata
I20260620 20:18:16.788739 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet 483a43802c914326a77fab974c724186 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:16 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 20:18:16.788899 5306 tablet_replica.cc:330] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: stopping tablet replica
I20260620 20:18:16.789019 5306 raft_consensus.cc:2229] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:16.789127 5306 raft_consensus.cc:2258] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:16.789206 4704 catalog_manager.cc:4841] TS 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465): tablet 483a43802c914326a77fab974c724186 (table test-table [id=673f54780c6d42b0b38d2325080fedeb]) successfully deleted
I20260620 20:18:16.789746 5306 ts_tablet_manager.cc:1905] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:16.790786 5135 tablet_service.cc:1467] Processing CreateTablet for tablet e7fbc69343e94c998a2c53d2b0c12160 (DEFAULT_TABLE table=test-table [id=f5f9dcd8bc864ee1a20a345c59f3f0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:16.790848 4995 tablet_service.cc:1467] Processing CreateTablet for tablet e7fbc69343e94c998a2c53d2b0c12160 (DEFAULT_TABLE table=test-table [id=f5f9dcd8bc864ee1a20a345c59f3f0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:16.791021 5135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7fbc69343e94c998a2c53d2b0c12160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:16.791115 4995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7fbc69343e94c998a2c53d2b0c12160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:16.791985 5306 ts_tablet_manager.cc:1918] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:16.792052 5306 log.cc:1196] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/483a43802c914326a77fab974c724186
I20260620 20:18:16.792280 5306 ts_tablet_manager.cc:1939] T 483a43802c914326a77fab974c724186 P 4a18f0089fdb459799a0ec663eb81796: Deleting consensus metadata
I20260620 20:18:16.792974 5319 tablet_bootstrap.cc:492] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Bootstrap starting.
I20260620 20:18:16.793104 4705 catalog_manager.cc:4841] TS 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271): tablet 483a43802c914326a77fab974c724186 (table test-table [id=673f54780c6d42b0b38d2325080fedeb]) successfully deleted
I20260620 20:18:16.793790 5319 tablet_bootstrap.cc:654] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:16.793906 4705 catalog_manager.cc:4841] TS 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785): tablet 483a43802c914326a77fab974c724186 (table test-table [id=673f54780c6d42b0b38d2325080fedeb]) successfully deleted
I20260620 20:18:16.794260 4854 tablet_service.cc:1467] Processing CreateTablet for tablet e7fbc69343e94c998a2c53d2b0c12160 (DEFAULT_TABLE table=test-table [id=f5f9dcd8bc864ee1a20a345c59f3f0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:16.794492 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7fbc69343e94c998a2c53d2b0c12160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:16.794741 5319 tablet_bootstrap.cc:492] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: No bootstrap required, opened a new log
I20260620 20:18:16.794795 5319 ts_tablet_manager.cc:1397] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:16.795102 5319 raft_consensus.cc:348] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.795204 5319 raft_consensus.cc:374] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:16.795235 5319 raft_consensus.cc:729] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a18f0089fdb459799a0ec663eb81796, State: Initialized, Role: FOLLOWER
I20260620 20:18:16.795363 5319 consensus_queue.cc:260] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.795619 5319 ts_tablet_manager.cc:1428] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:16.796599 5316 tablet_bootstrap.cc:492] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81: Bootstrap starting.
I20260620 20:18:16.797201 5317 tablet_bootstrap.cc:492] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Bootstrap starting.
I20260620 20:18:16.797538 5316 tablet_bootstrap.cc:654] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:16.798171 5317 tablet_bootstrap.cc:654] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:16.798655 5316 tablet_bootstrap.cc:492] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81: No bootstrap required, opened a new log
I20260620 20:18:16.798748 5316 ts_tablet_manager.cc:1397] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:16.799119 5316 raft_consensus.cc:348] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.799211 5316 raft_consensus.cc:374] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:16.799249 5316 raft_consensus.cc:729] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae93d0542a4e6db3acbd0c74016e81, State: Initialized, Role: FOLLOWER
I20260620 20:18:16.799355 5316 consensus_queue.cc:260] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.799546 5317 tablet_bootstrap.cc:492] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: No bootstrap required, opened a new log
I20260620 20:18:16.799594 5316 ts_tablet_manager.cc:1428] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:16.799623 5317 ts_tablet_manager.cc:1397] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:18:16.800035 5317 raft_consensus.cc:348] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.800132 5317 raft_consensus.cc:374] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:16.800161 5317 raft_consensus.cc:729] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Initialized, Role: FOLLOWER
I20260620 20:18:16.800258 5317 consensus_queue.cc:260] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.800518 5317 ts_tablet_manager.cc:1428] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:18:16.857748 5322 raft_consensus.cc:482] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:16.857867 5322 raft_consensus.cc:504] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.858263 5322 leader_election.cc:288] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785), 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:16.858634 5154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7fbc69343e94c998a2c53d2b0c12160" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796" is_pre_election: true
I20260620 20:18:16.858646 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7fbc69343e94c998a2c53d2b0c12160" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fae93d0542a4e6db3acbd0c74016e81" is_pre_election: true
I20260620 20:18:16.858760 4875 raft_consensus.cc:2454] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 0.
I20260620 20:18:16.858768 5154 raft_consensus.cc:2454] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 0.
I20260620 20:18:16.859025 4949 leader_election.cc:302] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a18f0089fdb459799a0ec663eb81796, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:16.859171 5322 raft_consensus.cc:2781] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:16.859238 5322 raft_consensus.cc:482] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:16.859283 5322 raft_consensus.cc:3037] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:16.860149 5322 raft_consensus.cc:504] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.860453 5322 leader_election.cc:288] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 election: Requested vote from peers 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785), 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271)
I20260620 20:18:16.860716 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7fbc69343e94c998a2c53d2b0c12160" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fae93d0542a4e6db3acbd0c74016e81"
I20260620 20:18:16.860790 5154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7fbc69343e94c998a2c53d2b0c12160" candidate_uuid: "91db6e06089347b6a6fbc888a3693f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a18f0089fdb459799a0ec663eb81796"
I20260620 20:18:16.860842 4875 raft_consensus.cc:3037] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:16.860893 5154 raft_consensus.cc:3037] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:16.861718 4875 raft_consensus.cc:2454] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 1.
I20260620 20:18:16.861943 5154 raft_consensus.cc:2454] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91db6e06089347b6a6fbc888a3693f1f in term 1.
I20260620 20:18:16.862056 4949 leader_election.cc:302] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fae93d0542a4e6db3acbd0c74016e81, 91db6e06089347b6a6fbc888a3693f1f; no voters:
I20260620 20:18:16.862362 5322 raft_consensus.cc:2781] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:16.862448 5322 raft_consensus.cc:686] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 1 LEADER]: Becoming Leader. State: Replica: 91db6e06089347b6a6fbc888a3693f1f, State: Running, Role: LEADER
I20260620 20:18:16.862572 5322 consensus_queue.cc:237] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } }
I20260620 20:18:16.864133 4718 catalog_manager.cc:5495] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194). New cstate: current_term: 1 leader_uuid: "91db6e06089347b6a6fbc888a3693f1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91db6e06089347b6a6fbc888a3693f1f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:16.884922 5155 raft_consensus.cc:1261] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Refusing update from remote peer 91db6e06089347b6a6fbc888a3693f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:16.885310 4875 raft_consensus.cc:1261] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Refusing update from remote peer 91db6e06089347b6a6fbc888a3693f1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:16.885533 5336 consensus_queue.cc:1035] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a18f0089fdb459799a0ec663eb81796" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:16.885674 5323 consensus_queue.cc:1035] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fae93d0542a4e6db3acbd0c74016e81" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:18:16.898249 5288 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:18:18.457830 4717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:18.458035 4717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:18.460325 4717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eca5971138e40209e364e109f1cba86: Sending DeleteTablet for 3 replicas of tablet e7fbc69343e94c998a2c53d2b0c12160
I20260620 20:18:18.460821 4854 tablet_service.cc:1514] Processing DeleteTablet for tablet e7fbc69343e94c998a2c53d2b0c12160 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:18 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:36494
I20260620 20:18:18.460852 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet e7fbc69343e94c998a2c53d2b0c12160 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:18 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 20:18:18.460870 4995 tablet_service.cc:1514] Processing DeleteTablet for tablet e7fbc69343e94c998a2c53d2b0c12160 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:18 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:59568
I20260620 20:18:18.461140 5367 tablet_replica.cc:330] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: stopping tablet replica
I20260620 20:18:18.461190 5369 tablet_replica.cc:330] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: stopping tablet replica
I20260620 20:18:18.461290 5368 tablet_replica.cc:330] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81: stopping tablet replica
I20260620 20:18:18.461323 5367 raft_consensus.cc:2229] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:18.461406 5369 raft_consensus.cc:2229] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:18.461428 5368 raft_consensus.cc:2229] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:18.461454 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4790
I20260620 20:18:18.461475 5367 raft_consensus.cc:2258] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:18.461526 5368 raft_consensus.cc:2258] T e7fbc69343e94c998a2c53d2b0c12160 P 6fae93d0542a4e6db3acbd0c74016e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:18.462275 5369 raft_consensus.cc:2258] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:18.462778 5369 ts_tablet_manager.cc:1905] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:18.462783 5367 ts_tablet_manager.cc:1905] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:18.465392 5369 ts_tablet_manager.cc:1918] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:18:18.465392 5367 ts_tablet_manager.cc:1918] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:18:18.465477 5367 log.cc:1196] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/e7fbc69343e94c998a2c53d2b0c12160
I20260620 20:18:18.465487 5369 log.cc:1196] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/e7fbc69343e94c998a2c53d2b0c12160
I20260620 20:18:18.465793 5369 ts_tablet_manager.cc:1939] T e7fbc69343e94c998a2c53d2b0c12160 P 91db6e06089347b6a6fbc888a3693f1f: Deleting consensus metadata
I20260620 20:18:18.465808 5367 ts_tablet_manager.cc:1939] T e7fbc69343e94c998a2c53d2b0c12160 P 4a18f0089fdb459799a0ec663eb81796: Deleting consensus metadata
I20260620 20:18:18.466450 4704 catalog_manager.cc:4841] TS 91db6e06089347b6a6fbc888a3693f1f (127.2.235.194:42465): tablet e7fbc69343e94c998a2c53d2b0c12160 (table test-table [id=f5f9dcd8bc864ee1a20a345c59f3f0f8]) successfully deleted
I20260620 20:18:18.466496 4705 catalog_manager.cc:4841] TS 4a18f0089fdb459799a0ec663eb81796 (127.2.235.195:45271): tablet e7fbc69343e94c998a2c53d2b0c12160 (table test-table [id=f5f9dcd8bc864ee1a20a345c59f3f0f8]) successfully deleted
W20260620 20:18:18.470333 4705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:39785 (error 108)
I20260620 20:18:18.470546 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4930
I20260620 20:18:18.472110 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:18.471646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39785 (local address 127.0.0.1:39536)
0620 20:18:18.471960 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:18.472040 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:39785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
W20260620 20:18:18.472240 4705 catalog_manager.cc:4584] TS 6fae93d0542a4e6db3acbd0c74016e81 (127.2.235.193:39785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7fbc69343e94c998a2c53d2b0c12160: Network error: Client connection negotiation failed: client connection to 127.2.235.193:39785: connect: Connection refused (error 111)
I20260620 20:18:18.477504 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 5070
I20260620 20:18:18.483907 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 4686
2026-06-20T20:18:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5742 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:18 dist-test-slave-8z53 krb5kdc[5374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:18 dist-test-slave-8z53 krb5kdc[5374](info): set up 2 sockets
Jun 20 20:18:18 dist-test-slave-8z53 krb5kdc[5374](info): commencing operation
krb5kdc: starting...
W20260620 20:18:19.469123 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:19 dist-test-slave-8z53 krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:19.572068 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33301
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:33301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:19.668839 5387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:19.669054 5387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:19.669088 5387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:19.672931 5387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:19.672989 5387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:19.673007 5387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:19.673022 5387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:19.673067 5387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:19.677964 5387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:33301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:19.678880 5387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:19.679798 5387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:19.687990 5393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:19.687988 5392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:19.687985 5395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:19.688330 5387 server_base.cc:1034] running on GCE node
I20260620 20:18:19.688925 5387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:19.690047 5387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:19.691234 5387 hybrid_clock.cc:648] HybridClock initialized: now 1781986699691212 us; error 39 us; skew 500 ppm
Jun 20 20:18:19 dist-test-slave-8z53 krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986699, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:19.698881 5387 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:19.699693 5387 webserver.cc:460] Webserver started at http://127.2.235.254:36531/ using document root <none> and password file <none>
I20260620 20:18:19.700217 5387 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:19.700302 5387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:19.700516 5387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:19.702541 5387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "b7e1ab251582437cb797d2bb6b6d0eb8"
format_stamp: "Formatted at 2026-06-20 20:18:19 on dist-test-slave-8z53"
server_key: "37ec5bf8f8d9b8b29cc7daa62dfe4976"
server_key_iv: "538df60aad8bbf95acf78edf44817b72"
server_key_version: "encryptionkey@0"
I20260620 20:18:19.703105 5387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "b7e1ab251582437cb797d2bb6b6d0eb8"
format_stamp: "Formatted at 2026-06-20 20:18:19 on dist-test-slave-8z53"
server_key: "37ec5bf8f8d9b8b29cc7daa62dfe4976"
server_key_iv: "538df60aad8bbf95acf78edf44817b72"
server_key_version: "encryptionkey@0"
I20260620 20:18:19.706719 5387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:19.708954 5402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:19.710134 5387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:19.710259 5387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "b7e1ab251582437cb797d2bb6b6d0eb8"
format_stamp: "Formatted at 2026-06-20 20:18:19 on dist-test-slave-8z53"
server_key: "37ec5bf8f8d9b8b29cc7daa62dfe4976"
server_key_iv: "538df60aad8bbf95acf78edf44817b72"
server_key_version: "encryptionkey@0"
I20260620 20:18:19.710386 5387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:19.729033 5387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:19.732265 5387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:19.732407 5387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:19.739542 5387 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:33301
I20260620 20:18:19.739578 5454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:33301 every 8 connection(s)
I20260620 20:18:19.740429 5387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:19.740796 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 5387
I20260620 20:18:19.740909 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:19.741148 2991 external_mini_cluster.cc:1428] Setting key 1dc671d2d2f39298b6edf08c07d4635c
I20260620 20:18:19.743170 5455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:19.749591 5455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8: Bootstrap starting.
Jun 20 20:18:19 dist-test-slave-8z53 krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:19.751896 5455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:19.752614 5455 log.cc:824] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:19.754503 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:19.742432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.2.235.254:33301)
0620 20:18:19.742813 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:19.742827 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:19.742858 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:18:19.743478 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:19.743493 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:19.743774 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:19.744025 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:19.744961 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:19.745978 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:19.747188 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:19.747614 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:19.747714 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:19.750243 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:19.750247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:19.750256 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:19.750295 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:19.751772 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:19.752263 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:19.752269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:19.752272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:19.752370 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:19.752647 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:19.752651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:19.752653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:19.753128 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:19.753341 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:19.753618 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 20:18:19.753865 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":136,"threads_started":1}
I20260620 20:18:19.754528 5455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8: No bootstrap required, opened a new log
I20260620 20:18:19.758368 5455 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33301 } }
I20260620 20:18:19.758613 5455 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:19.758694 5455 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e1ab251582437cb797d2bb6b6d0eb8, State: Initialized, Role: FOLLOWER
I20260620 20:18:19.759136 5455 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33301 } }
I20260620 20:18:19.759282 5455 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:19.759371 5455 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:19.759471 5455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:19.760358 5455 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33301 } }
I20260620 20:18:19.760713 5455 leader_election.cc:302] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e1ab251582437cb797d2bb6b6d0eb8; no voters:
I20260620 20:18:19.760977 5455 leader_election.cc:288] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:19.761094 5460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:19.761379 5460 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [term 1 LEADER]: Becoming Leader. State: Replica: b7e1ab251582437cb797d2bb6b6d0eb8, State: Running, Role: LEADER
I20260620 20:18:19.761709 5460 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33301 } }
I20260620 20:18:19.762140 5455 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:19.763263 5462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e1ab251582437cb797d2bb6b6d0eb8. Latest consensus state: current_term: 1 leader_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33301 } } }
I20260620 20:18:19.763387 5462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:19.763701 5461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e1ab251582437cb797d2bb6b6d0eb8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33301 } } }
I20260620 20:18:19.763798 5461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:19.764544 5469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:19.767143 5469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:19.772190 5469 catalog_manager.cc:1347] Generated new cluster ID: 044f532013d740d9bbeccaf581fb1b8c
I20260620 20:18:19.772246 5469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:19.782699 5469 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:19.783411 5469 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:19.790320 5469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e1ab251582437cb797d2bb6b6d0eb8: Generated new TSK 0
I20260620 20:18:19.790885 5469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:19.819365 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33301
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:19.915879 5481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:19.916100 5481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:19.916169 5481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:19.919806 5481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:19.919875 5481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:19.919951 5481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:19.924714 5481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:19.925874 5481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:19.926870 5481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:19.935724 5486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:19.935730 5489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:19.935909 5481 server_base.cc:1034] running on GCE node
W20260620 20:18:19.935751 5487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:19.936401 5481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:19.937006 5481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:19.938197 5481 hybrid_clock.cc:648] HybridClock initialized: now 1781986699938161 us; error 46 us; skew 500 ppm
Jun 20 20:18:19 dist-test-slave-8z53 krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986699, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:19.940811 5481 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:19.941619 5481 webserver.cc:460] Webserver started at http://127.2.235.193:43783/ using document root <none> and password file <none>
I20260620 20:18:19.942082 5481 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:19.942134 5481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:19.942279 5481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:19.944074 5481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "b3aab8a9227d408987e366e08d883207"
format_stamp: "Formatted at 2026-06-20 20:18:19 on dist-test-slave-8z53"
server_key: "65c0b20b3ce8b276e6a270dee10c55f4"
server_key_iv: "742e8b138bcc2cbd2fa7ef70b85a8a8c"
server_key_version: "encryptionkey@0"
I20260620 20:18:19.944561 5481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "b3aab8a9227d408987e366e08d883207"
format_stamp: "Formatted at 2026-06-20 20:18:19 on dist-test-slave-8z53"
server_key: "65c0b20b3ce8b276e6a270dee10c55f4"
server_key_iv: "742e8b138bcc2cbd2fa7ef70b85a8a8c"
server_key_version: "encryptionkey@0"
I20260620 20:18:19.947830 5481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:18:19.949831 5496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:19.950893 5481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:19.951062 5481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "b3aab8a9227d408987e366e08d883207"
format_stamp: "Formatted at 2026-06-20 20:18:19 on dist-test-slave-8z53"
server_key: "65c0b20b3ce8b276e6a270dee10c55f4"
server_key_iv: "742e8b138bcc2cbd2fa7ef70b85a8a8c"
server_key_version: "encryptionkey@0"
I20260620 20:18:19.951149 5481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:19.960248 5481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:19.962734 5481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:19.962930 5481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:19.963510 5481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:19.964171 5481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:19.964210 5481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:19.964242 5481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:19.964264 5481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:19.971906 5481 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:39957
I20260620 20:18:19.971930 5609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:39957 every 8 connection(s)
I20260620 20:18:19.972630 5481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:19.974176 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 5481
I20260620 20:18:19.974287 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:19.974540 2991 external_mini_cluster.cc:1428] Setting key 4fea982116c2985ccc885af4cb267fde
Jun 20 20:18:19 dist-test-slave-8z53 krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986699, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:18:19.986142 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:19.974182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50757 (local address 127.2.235.254:33301)
0620 20:18:19.974339 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:19.974343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:19.975041 (+ 698us) server_negotiation.cc:407] Connection header received
0620 20:18:19.976321 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:19.976327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:19.976394 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:19.976493 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:19.977488 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:19.977967 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:19.979493 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:19.979787 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:19.979900 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:19.982608 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:19.982612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:19.982613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:19.982636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:19.984034 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:19.984620 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:19.984622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:19.984624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:19.984696 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:19.985063 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:19.985065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:19.985066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:19.985217 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:19.985354 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:19.985934 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 20:18:19.986037 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:18:19.987233 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:19.974479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33301 (local address 127.2.235.193:50757)
0620 20:18:19.974901 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:19.974947 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:19.976140 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:19.976617 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:19.976622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:19.976904 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:19.977358 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:19.977372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:19.978075 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:19.978078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:19.979373 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:19.979386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:19.979984 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:19.979988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:19.980209 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:19.980763 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:19.980789 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:19.982417 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:19.984150 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:19.984154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:19.984163 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:19.984525 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:19.984784 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:19.984787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:19.984788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:19.984918 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:19.985368 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:19.985373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:19.985652 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 20:18:19.985883 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 20:18:19.986218 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260620 20:18:19.988350 5610 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33301
I20260620 20:18:19.988585 5610 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:19.989012 5610 heartbeater.cc:511] Master 127.2.235.254:33301 requested a full tablet report, sending...
I20260620 20:18:19.990397 5419 ts_manager.cc:194] Registered new tserver with Master: b3aab8a9227d408987e366e08d883207 (127.2.235.193:39957)
I20260620 20:18:19.991374 5419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:50757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:19.999213 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33301
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:20.094419 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:20.094653 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:20.094692 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:20.098312 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:20.098353 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:20.098402 5615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:20.102763 5615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:20.103551 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:20.104470 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:20.112640 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:20.112692 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:20.112643 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:20.112983 5615 server_base.cc:1034] running on GCE node
I20260620 20:18:20.113368 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:20.113808 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:20.114955 5615 hybrid_clock.cc:648] HybridClock initialized: now 1781986700114934 us; error 30 us; skew 500 ppm
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:20.117475 5615 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:20.118180 5615 webserver.cc:460] Webserver started at http://127.2.235.194:38371/ using document root <none> and password file <none>
I20260620 20:18:20.118664 5615 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:20.118728 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:20.118928 5615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:20.120568 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "385eac5526eb453da19a478aea353526"
format_stamp: "Formatted at 2026-06-20 20:18:20 on dist-test-slave-8z53"
server_key: "cf992a787398ffa1e3e7b4523ebf006d"
server_key_iv: "473e118601679c9873edfe6cb295b369"
server_key_version: "encryptionkey@0"
I20260620 20:18:20.121061 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "385eac5526eb453da19a478aea353526"
format_stamp: "Formatted at 2026-06-20 20:18:20 on dist-test-slave-8z53"
server_key: "cf992a787398ffa1e3e7b4523ebf006d"
server_key_iv: "473e118601679c9873edfe6cb295b369"
server_key_version: "encryptionkey@0"
I20260620 20:18:20.124380 5615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:18:20.126670 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:20.127698 5615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:20.127820 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "385eac5526eb453da19a478aea353526"
format_stamp: "Formatted at 2026-06-20 20:18:20 on dist-test-slave-8z53"
server_key: "cf992a787398ffa1e3e7b4523ebf006d"
server_key_iv: "473e118601679c9873edfe6cb295b369"
server_key_version: "encryptionkey@0"
I20260620 20:18:20.127912 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:20.138543 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:20.140799 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:20.140975 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:20.141507 5615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:20.142194 5615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:20.142257 5615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:20.142318 5615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:20.142357 5615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:20.150177 5615 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:45659
I20260620 20:18:20.150208 5743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:45659 every 8 connection(s)
I20260620 20:18:20.150853 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:20.153563 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 5615
I20260620 20:18:20.153671 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:20.153899 2991 external_mini_cluster.cc:1428] Setting key e5b3005259b2d58bc9cd9e7814952a47
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:18:20.163576 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:20.152256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:44957 (local address 127.2.235.254:33301)
0620 20:18:20.152373 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:20.152376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:20.153002 (+ 626us) server_negotiation.cc:407] Connection header received
0620 20:18:20.154605 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:20.154608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:20.154652 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:20.154737 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:20.155964 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.156438 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.157512 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.157709 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.157808 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:20.160097 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:20.160100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:20.160102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:20.160122 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:20.161569 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:20.162161 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:20.162164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:20.162165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:20.162235 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:20.162586 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:20.162588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:20.162589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:20.162769 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:20.162891 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:20.163355 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 20:18:20.163453 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:18:20.164253 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:20.152527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33301 (local address 127.2.235.194:44957)
0620 20:18:20.152879 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:20.152912 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:20.154214 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:20.154872 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:20.154878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:20.155362 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:20.155822 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:20.155834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.156554 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.156559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:20.157405 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:20.157413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.157831 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.157835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:20.157989 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:20.158457 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:20.158469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:20.159945 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:20.161672 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:20.161679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:20.161692 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:20.162050 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:20.162323 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:20.162326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:20.162328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:20.162501 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:20.162855 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:20.162859 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:20.163142 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 20:18:20.163301 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 20:18:20.163510 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:18:20.165418 5744 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33301
I20260620 20:18:20.165654 5744 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:20.166132 5744 heartbeater.cc:511] Master 127.2.235.254:33301 requested a full tablet report, sending...
I20260620 20:18:20.167081 5419 ts_manager.cc:194] Registered new tserver with Master: 385eac5526eb453da19a478aea353526 (127.2.235.194:45659)
I20260620 20:18:20.167570 5419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:44957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:20.177549 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33301
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:20.275951 5749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:20.276144 5749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:20.276180 5749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:20.279613 5749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:20.279654 5749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:20.279702 5749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:20.284277 5749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:20.285086 5749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:20.285903 5749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:20.294183 5755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:20.294189 5754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:20.294286 5749 server_base.cc:1034] running on GCE node
W20260620 20:18:20.294194 5757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:20.294783 5749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:20.295341 5749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:20.296593 5749 hybrid_clock.cc:648] HybridClock initialized: now 1781986700296541 us; error 23 us; skew 500 ppm
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:20.299121 5749 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:20.299890 5749 webserver.cc:460] Webserver started at http://127.2.235.195:35793/ using document root <none> and password file <none>
I20260620 20:18:20.300377 5749 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:20.300457 5749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:20.300661 5749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:20.302443 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "d522d0ca14714cfb90e29b315f894bd7"
format_stamp: "Formatted at 2026-06-20 20:18:20 on dist-test-slave-8z53"
server_key: "7d86a70e0f94fc38ac299e2e1b8dae6a"
server_key_iv: "c544e38900dc8ed07ae470a669fbaa37"
server_key_version: "encryptionkey@0"
I20260620 20:18:20.302971 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "d522d0ca14714cfb90e29b315f894bd7"
format_stamp: "Formatted at 2026-06-20 20:18:20 on dist-test-slave-8z53"
server_key: "7d86a70e0f94fc38ac299e2e1b8dae6a"
server_key_iv: "c544e38900dc8ed07ae470a669fbaa37"
server_key_version: "encryptionkey@0"
I20260620 20:18:20.306277 5749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:18:20.308248 5764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:20.309374 5749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:20.309491 5749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "d522d0ca14714cfb90e29b315f894bd7"
format_stamp: "Formatted at 2026-06-20 20:18:20 on dist-test-slave-8z53"
server_key: "7d86a70e0f94fc38ac299e2e1b8dae6a"
server_key_iv: "c544e38900dc8ed07ae470a669fbaa37"
server_key_version: "encryptionkey@0"
I20260620 20:18:20.309602 5749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:20.319355 5749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:20.321879 5749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:20.322068 5749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:20.322556 5749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:20.323217 5749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:20.323285 5749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:20.323346 5749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:20.323390 5749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:20.331473 5749 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:36825
I20260620 20:18:20.331509 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:36825 every 8 connection(s)
I20260620 20:18:20.332275 5749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986700, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:20.342294 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 5749
I20260620 20:18:20.342403 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:20.342733 2991 external_mini_cluster.cc:1428] Setting key 57ac8d2425bed6128603b40431a78440
I20260620 20:18:20.344954 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:20.333925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:36711 (local address 127.2.235.254:33301)
0620 20:18:20.334048 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:20.334052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:20.334763 (+ 711us) server_negotiation.cc:407] Connection header received
0620 20:18:20.336015 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:20.336018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:20.336102 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:20.336203 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:20.337241 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.337814 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.338906 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.339126 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.339233 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:20.341382 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:20.341387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:20.341396 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:20.341418 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:20.342756 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:20.343365 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:20.343369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:20.343370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:20.343448 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:20.343835 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:20.343838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:20.343840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:20.344062 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:20.344156 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:20.344696 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 20:18:20.344833 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:18:20.345690 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:20.334237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33301 (local address 127.2.235.195:36711)
0620 20:18:20.334630 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:20.334667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:20.335823 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:20.336363 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:20.336369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:20.336664 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:20.337091 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:20.337103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.337948 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.337951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:20.338757 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:20.338766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:20.339264 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:20.339268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:20.339422 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:20.339864 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:20.339878 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:20.341186 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:20.342899 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:20.342903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:20.342912 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:20.343253 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:20.343573 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:20.343578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:20.343583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:20.343740 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:20.344179 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:20.344184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:20.344432 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 20:18:20.344647 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 20:18:20.344909 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
I20260620 20:18:20.346587 5878 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33301
I20260620 20:18:20.346782 5878 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:20.347182 5878 heartbeater.cc:511] Master 127.2.235.254:33301 requested a full tablet report, sending...
I20260620 20:18:20.348024 5419 ts_manager.cc:194] Registered new tserver with Master: d522d0ca14714cfb90e29b315f894bd7 (127.2.235.195:36825)
I20260620 20:18:20.348508 5419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:36711
I20260620 20:18:20.356240 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:18:20.367460 5458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:20.366383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.2.235.254:33301)
0620 20:18:20.366479 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:20.366482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:20.366669 (+ 187us) server_negotiation.cc:407] Connection header received
0620 20:18:20.366750 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:20.366752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:20.366783 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:20.366842 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:20.367348 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51878: BlockingRecv error: recv got EOF from 127.0.0.1:51878 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:18:20.368417 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 5481
I20260620 20:18:20.373698 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 5615
I20260620 20:18:20.378763 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 5749
I20260620 20:18:20.383911 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 5387
2026-06-20T20:18:20Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1897 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5890](info): set up 2 sockets
Jun 20 20:18:20 dist-test-slave-8z53 krb5kdc[5890](info): commencing operation
krb5kdc: starting...
W20260620 20:18:21.357164 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:21 dist-test-slave-8z53 krb5kdc[5890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:21.463286 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40057
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:40057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:21.561528 5903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:21.561748 5903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:21.561784 5903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:21.565781 5903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:21.565840 5903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:21.565857 5903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:21.565872 5903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:21.565917 5903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:21.570298 5903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:40057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:21.571134 5903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:21.571951 5903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:21.579487 5911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:21.579741 5903 server_base.cc:1034] running on GCE node
W20260620 20:18:21.579499 5909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:21.579499 5908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:21.580376 5903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:21.581331 5903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:21.582494 5903 hybrid_clock.cc:648] HybridClock initialized: now 1781986701582470 us; error 35 us; skew 500 ppm
Jun 20 20:18:21 dist-test-slave-8z53 krb5kdc[5890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986701, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:21.584949 5903 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:21.585788 5903 webserver.cc:460] Webserver started at http://127.2.235.254:39555/ using document root <none> and password file <none>
I20260620 20:18:21.586298 5903 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:21.586383 5903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:21.586606 5903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:21.588339 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "7df3963008ef4d688b8b3a7133ff446d"
format_stamp: "Formatted at 2026-06-20 20:18:21 on dist-test-slave-8z53"
server_key: "f692d362f63e8850aae687dc0fb9f007"
server_key_iv: "32301675340866d5c5f0ee5db9d1829f"
server_key_version: "encryptionkey@0"
I20260620 20:18:21.588845 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "7df3963008ef4d688b8b3a7133ff446d"
format_stamp: "Formatted at 2026-06-20 20:18:21 on dist-test-slave-8z53"
server_key: "f692d362f63e8850aae687dc0fb9f007"
server_key_iv: "32301675340866d5c5f0ee5db9d1829f"
server_key_version: "encryptionkey@0"
I20260620 20:18:21.592054 5903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:18:21.594156 5918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:21.595263 5903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:21.595402 5903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "7df3963008ef4d688b8b3a7133ff446d"
format_stamp: "Formatted at 2026-06-20 20:18:21 on dist-test-slave-8z53"
server_key: "f692d362f63e8850aae687dc0fb9f007"
server_key_iv: "32301675340866d5c5f0ee5db9d1829f"
server_key_version: "encryptionkey@0"
I20260620 20:18:21.595516 5903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:21.611866 5903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:21.614439 5903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:21.614629 5903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:21.620766 5903 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:40057
I20260620 20:18:21.621080 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:40057 every 8 connection(s)
I20260620 20:18:21.621628 5903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:21.624119 5971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:21.627785 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 5903
I20260620 20:18:21.627905 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:21.628154 2991 external_mini_cluster.cc:1428] Setting key dcb8f948dc14a27a80ccadf62593da2d
I20260620 20:18:21.629591 5971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d: Bootstrap starting.
I20260620 20:18:21.632157 5971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:21.633040 5971 log.cc:824] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:21.635094 5971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d: No bootstrap required, opened a new log
Jun 20 20:18:21 dist-test-slave-8z53 krb5kdc[5890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:21.639032 5971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3963008ef4d688b8b3a7133ff446d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40057 } }
I20260620 20:18:21.639281 5971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:21.639356 5971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df3963008ef4d688b8b3a7133ff446d, State: Initialized, Role: FOLLOWER
I20260620 20:18:21.639788 5971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3963008ef4d688b8b3a7133ff446d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40057 } }
I20260620 20:18:21.639930 5971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:21.640005 5971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:21.640101 5971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:21.640988 5971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3963008ef4d688b8b3a7133ff446d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40057 } }
I20260620 20:18:21.641335 5971 leader_election.cc:302] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df3963008ef4d688b8b3a7133ff446d; no voters:
I20260620 20:18:21.641657 5971 leader_election.cc:288] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:21.641798 5976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:21.641965 5976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [term 1 LEADER]: Becoming Leader. State: Replica: 7df3963008ef4d688b8b3a7133ff446d, State: Running, Role: LEADER
I20260620 20:18:21.642222 5976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3963008ef4d688b8b3a7133ff446d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40057 } }
I20260620 20:18:21.642608 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:21.629398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.2.235.254:40057)
0620 20:18:21.629787 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:21.629807 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:21.629848 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:18:21.630651 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:21.630675 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:21.631062 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:21.631400 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:21.632042 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:21.633519 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:21.634430 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:21.635111 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:21.635288 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:21.637999 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:21.638005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:21.638020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:21.638060 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:21.639872 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:21.640351 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:21.640356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:21.640359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:21.640456 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:21.640714 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:21.640716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:21.640717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:21.641018 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:21.641226 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:21.641506 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 20:18:21.641774 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
I20260620 20:18:21.642722 5971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:21.643219 5976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df3963008ef4d688b8b3a7133ff446d. Latest consensus state: current_term: 1 leader_uuid: "7df3963008ef4d688b8b3a7133ff446d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3963008ef4d688b8b3a7133ff446d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40057 } } }
I20260620 20:18:21.643308 5976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:21.643580 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df3963008ef4d688b8b3a7133ff446d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df3963008ef4d688b8b3a7133ff446d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40057 } } }
I20260620 20:18:21.643664 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:21.645123 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:18:21.644619 5981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:21.647920 5981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:21.653045 5981 catalog_manager.cc:1347] Generated new cluster ID: e0bb423ced47479dbddd7ecd45453ff8
I20260620 20:18:21.653096 5981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:21.667590 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:21.662537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.2.235.254:40057)
0620 20:18:21.662620 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:21.662624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:21.662846 (+ 222us) server_negotiation.cc:407] Connection header received
0620 20:18:21.662945 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:21.662948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:21.662996 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:21.663072 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:21.663522 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:21.664290 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:21.665391 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:21.665756 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:21.665879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:21.666641 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:21.666645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:21.666648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:21.667153 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:21.667185 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:21.667204 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:21.667293 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:21.667383 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 20:18:21.667441 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":30}
I20260620 20:18:21.673338 5981 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:21.674458 5981 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:21.685604 5981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df3963008ef4d688b8b3a7133ff446d: Generated new TSK 0
I20260620 20:18:21.686326 5981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:21.695101 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 5903
2026-06-20T20:18:21Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1309 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:21 dist-test-slave-8z53 krb5kdc[6004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:21 dist-test-slave-8z53 krb5kdc[6004](info): set up 2 sockets
Jun 20 20:18:21 dist-test-slave-8z53 krb5kdc[6004](info): commencing operation
krb5kdc: starting...
W20260620 20:18:22.645041 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:22 dist-test-slave-8z53 krb5kdc[6004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:22.748286 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44401
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:44401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:22.842953 6017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:22.843173 6017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:22.843238 6017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:22.846796 6017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:22.846865 6017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:22.846897 6017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:22.846959 6017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:22.846990 6017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:22.851163 6017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:44401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:22.851899 6017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:22.852658 6017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:22.859632 6022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:22.859666 6017 server_base.cc:1034] running on GCE node
W20260620 20:18:22.859654 6025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:22.859634 6023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:22.860235 6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:22.861079 6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:22.862244 6017 hybrid_clock.cc:648] HybridClock initialized: now 1781986702862227 us; error 36 us; skew 500 ppm
Jun 20 20:18:22 dist-test-slave-8z53 krb5kdc[6004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986702, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:22.864432 6017 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:22.865207 6017 webserver.cc:460] Webserver started at http://127.2.235.254:42333/ using document root <none> and password file <none>
I20260620 20:18:22.865708 6017 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:22.865795 6017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:22.866019 6017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:22.867583 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "7bf75341aa294ac0a397e80adbba4241"
format_stamp: "Formatted at 2026-06-20 20:18:22 on dist-test-slave-8z53"
server_key: "d4bbb1b7c403a4b4a5b0886e1c371ee6"
server_key_iv: "d03c9c0d825eafd1c44e537504992e30"
server_key_version: "encryptionkey@0"
I20260620 20:18:22.868060 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "7bf75341aa294ac0a397e80adbba4241"
format_stamp: "Formatted at 2026-06-20 20:18:22 on dist-test-slave-8z53"
server_key: "d4bbb1b7c403a4b4a5b0886e1c371ee6"
server_key_iv: "d03c9c0d825eafd1c44e537504992e30"
server_key_version: "encryptionkey@0"
I20260620 20:18:22.871037 6017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:18:22.872991 6032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:22.874047 6017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:22.874192 6017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "7bf75341aa294ac0a397e80adbba4241"
format_stamp: "Formatted at 2026-06-20 20:18:22 on dist-test-slave-8z53"
server_key: "d4bbb1b7c403a4b4a5b0886e1c371ee6"
server_key_iv: "d03c9c0d825eafd1c44e537504992e30"
server_key_version: "encryptionkey@0"
I20260620 20:18:22.874292 6017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:22.899010 6017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:22.901669 6017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:22.901858 6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:22.908843 6017 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:44401
I20260620 20:18:22.909113 6084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:44401 every 8 connection(s)
I20260620 20:18:22.909821 6017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:22.912477 6085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:22.912541 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6017
I20260620 20:18:22.912667 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:22.913107 2991 external_mini_cluster.cc:1428] Setting key fe919b9dee298e9e8f9aa244361d34cc
I20260620 20:18:22.919502 6085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241: Bootstrap starting.
I20260620 20:18:22.921721 6085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:22.922498 6085 log.cc:824] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:18:22 dist-test-slave-8z53 krb5kdc[6004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:22.924510 6085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241: No bootstrap required, opened a new log
I20260620 20:18:22.927104 6085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75341aa294ac0a397e80adbba4241" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44401 } }
I20260620 20:18:22.927371 6085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:22.927462 6085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf75341aa294ac0a397e80adbba4241, State: Initialized, Role: FOLLOWER
I20260620 20:18:22.927826 6085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75341aa294ac0a397e80adbba4241" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44401 } }
I20260620 20:18:22.927963 6085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:22.928040 6085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:22.928122 6085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:22.928965 6085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75341aa294ac0a397e80adbba4241" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44401 } }
I20260620 20:18:22.929317 6085 leader_election.cc:302] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bf75341aa294ac0a397e80adbba4241; no voters:
I20260620 20:18:22.929621 6085 leader_election.cc:288] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:22.929692 6090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:22.929857 6090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf75341aa294ac0a397e80adbba4241, State: Running, Role: LEADER
I20260620 20:18:22.930166 6090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75341aa294ac0a397e80adbba4241" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44401 } }
I20260620 20:18:22.930243 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:22.914442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.2.235.254:44401)
0620 20:18:22.914836 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:22.914857 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:22.916333 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 20:18:22.917239 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:22.917282 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:22.917722 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:22.918070 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:22.919004 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:22.920478 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:22.921566 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:22.922092 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:22.922216 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:22.924885 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:22.924891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:22.924903 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:22.924933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:22.927085 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:22.927604 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:22.927608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:22.927612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:22.927771 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:22.928114 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:22.928118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:22.928120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:22.928565 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:22.928716 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:22.929022 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:18:22.929341 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
I20260620 20:18:22.930778 6085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:22.931769 6091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bf75341aa294ac0a397e80adbba4241. Latest consensus state: current_term: 1 leader_uuid: "7bf75341aa294ac0a397e80adbba4241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75341aa294ac0a397e80adbba4241" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44401 } } }
I20260620 20:18:22.931902 6091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:22.932251 6092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bf75341aa294ac0a397e80adbba4241" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bf75341aa294ac0a397e80adbba4241" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44401 } } }
I20260620 20:18:22.932334 6092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:22.932520 6099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:22.935384 6099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:22.940289 6099 catalog_manager.cc:1347] Generated new cluster ID: c73188039ca94b9d85cdad3d5bbccea8
I20260620 20:18:22.940351 6099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:22.964815 6099 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:22.965726 6099 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:22.976308 6099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bf75341aa294ac0a397e80adbba4241: Generated new TSK 0
I20260620 20:18:22.976935 6099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:23.011865 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44401
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:23.108253 6111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:23.108518 6111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:23.108587 6111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:23.112139 6111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:23.112206 6111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:23.112280 6111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:23.116780 6111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:23.117559 6111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:23.118343 6111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:23.126602 6117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:23.126602 6116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:23.126698 6111 server_base.cc:1034] running on GCE node
W20260620 20:18:23.126654 6119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:23.127135 6111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:23.127746 6111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:23.128890 6111 hybrid_clock.cc:648] HybridClock initialized: now 1781986703128874 us; error 25 us; skew 500 ppm
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:23.131382 6111 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:23.132172 6111 webserver.cc:460] Webserver started at http://127.2.235.193:46509/ using document root <none> and password file <none>
I20260620 20:18:23.132581 6111 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:23.132632 6111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:23.132761 6111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:23.134517 6111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "dd7b794149094483aee74b27e3950d9e"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "f0c2c8ba4eff08f9d0b0afa89791e37e"
server_key_iv: "b51019fc493c4e4c8d7cd0c8e3e75355"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.134989 6111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "dd7b794149094483aee74b27e3950d9e"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "f0c2c8ba4eff08f9d0b0afa89791e37e"
server_key_iv: "b51019fc493c4e4c8d7cd0c8e3e75355"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.138181 6111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:23.140112 6126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.141117 6111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:23.141232 6111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "dd7b794149094483aee74b27e3950d9e"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "f0c2c8ba4eff08f9d0b0afa89791e37e"
server_key_iv: "b51019fc493c4e4c8d7cd0c8e3e75355"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.141369 6111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:23.150840 6111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:23.152994 6111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:23.153147 6111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:23.153615 6111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:23.154279 6111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:23.154341 6111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.154399 6111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:23.154443 6111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.162287 6111 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:36975
I20260620 20:18:23.162310 6239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:36975 every 8 connection(s)
I20260620 20:18:23.162966 6111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:23.166790 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6111
I20260620 20:18:23.166900 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:23.167191 2991 external_mini_cluster.cc:1428] Setting key dae8e29064d522d3fa9a8582bdbbc954
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:23.175802 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.164453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:47289 (local address 127.2.235.254:44401)
0620 20:18:23.164586 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:23.164591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:23.165390 (+ 799us) server_negotiation.cc:407] Connection header received
0620 20:18:23.166572 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:23.166574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:23.166616 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:23.166680 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:23.167884 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.168459 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.169507 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.169716 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.169816 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.172217 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:23.172220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:23.172221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:23.172240 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:23.173858 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.174361 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.174366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.174368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.174449 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.174784 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.174789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.174791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.174958 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:23.175077 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:23.175524 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 20:18:23.175637 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:23.176443 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.164815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44401 (local address 127.2.235.193:47289)
0620 20:18:23.165217 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:23.165303 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:23.166403 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:23.166839 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:23.166847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:23.167279 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:23.167745 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:23.167756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.168570 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.168573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:23.169398 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:23.169407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.169845 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.169849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:23.170009 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.170494 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:23.170508 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:23.172072 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:23.173982 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:23.173988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:23.174002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:23.174253 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:23.174550 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:23.174553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:23.174555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:23.174687 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:23.175088 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:23.175096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:23.175306 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 20:18:23.175488 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 20:18:23.175766 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260620 20:18:23.177628 6240 heartbeater.cc:347] Connected to a master server at 127.2.235.254:44401
I20260620 20:18:23.177857 6240 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:18:23.178395 6240 heartbeater.cc:511] Master 127.2.235.254:44401 requested a full tablet report, sending...
I20260620 20:18:23.179912 6049 ts_manager.cc:194] Registered new tserver with Master: dd7b794149094483aee74b27e3950d9e (127.2.235.193:36975)
I20260620 20:18:23.180985 6049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:47289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:23.191200 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44401
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:23.287117 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:23.287333 6245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:23.287400 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:23.290817 6245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:23.290884 6245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:23.290956 6245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:23.295305 6245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:23.296145 6245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:23.296980 6245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:23.305430 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:23.305452 6253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:23.305439 6251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:23.305507 6245 server_base.cc:1034] running on GCE node
I20260620 20:18:23.305945 6245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:23.306474 6245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:23.307628 6245 hybrid_clock.cc:648] HybridClock initialized: now 1781986703307611 us; error 26 us; skew 500 ppm
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:23.310144 6245 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:23.310914 6245 webserver.cc:460] Webserver started at http://127.2.235.194:43491/ using document root <none> and password file <none>
I20260620 20:18:23.311421 6245 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:23.311475 6245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:23.311677 6245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:23.313478 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "fbe722f11bb649ef9af9af9cb5f18a0a"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "97cff1e96bb40470e514e1f1bc498353"
server_key_iv: "2dc55aee92e765c4f138b40fadc2a98f"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.314021 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe722f11bb649ef9af9af9cb5f18a0a"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "97cff1e96bb40470e514e1f1bc498353"
server_key_iv: "2dc55aee92e765c4f138b40fadc2a98f"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.317286 6245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:23.319397 6260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.320451 6245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:23.320580 6245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "fbe722f11bb649ef9af9af9cb5f18a0a"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "97cff1e96bb40470e514e1f1bc498353"
server_key_iv: "2dc55aee92e765c4f138b40fadc2a98f"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.320680 6245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:23.342667 6245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:23.345136 6245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:23.345331 6245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:23.345844 6245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:23.346517 6245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:23.346561 6245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.346596 6245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:23.346616 6245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.354625 6245 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:34877
I20260620 20:18:23.354673 6373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:34877 every 8 connection(s)
I20260620 20:18:23.355388 6245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:23.355741 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6245
I20260620 20:18:23.355862 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:23.356174 2991 external_mini_cluster.cc:1428] Setting key bde5dbc3419e2e5acf3ecbdb9663a979
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:18:23.369573 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.356960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:53161 (local address 127.2.235.254:44401)
0620 20:18:23.357085 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:23.357089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:23.358014 (+ 925us) server_negotiation.cc:407] Connection header received
0620 20:18:23.359395 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:23.359397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:23.359437 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:23.359499 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:23.360872 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.361521 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.363259 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.363470 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.363537 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.366176 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:23.366179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:23.366181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:23.366202 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:23.367667 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.368160 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.368169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.368171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.368264 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.368580 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.368586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.368591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.368817 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:23.368914 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:23.369317 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 20:18:23.369421 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:18:23.370206 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.357409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44401 (local address 127.2.235.194:53161)
0620 20:18:23.357853 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:23.357907 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:23.359196 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:23.359653 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:23.359664 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:23.360125 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:23.360721 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:23.360740 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.361646 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.361651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:23.363144 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:23.363157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.363605 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.363610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:23.363812 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.364355 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:23.364373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:23.366005 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:23.367773 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:23.367778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:23.367789 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:23.368074 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:23.368356 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:23.368359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:23.368360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:23.368485 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:23.368909 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:23.368912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:23.369096 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 20:18:23.369254 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 20:18:23.369541 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":131,"threads_started":1}
I20260620 20:18:23.371271 6374 heartbeater.cc:347] Connected to a master server at 127.2.235.254:44401
I20260620 20:18:23.371488 6374 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:23.371896 6374 heartbeater.cc:511] Master 127.2.235.254:44401 requested a full tablet report, sending...
I20260620 20:18:23.372864 6049 ts_manager.cc:194] Registered new tserver with Master: fbe722f11bb649ef9af9af9cb5f18a0a (127.2.235.194:34877)
I20260620 20:18:23.373435 6049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:53161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:23.381512 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44401
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:23.483043 6379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:23.483232 6379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:23.483268 6379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:23.487001 6379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:23.487039 6379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:23.487085 6379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:23.491417 6379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:23.492149 6379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:23.492865 6379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:23.501070 6379 server_base.cc:1034] running on GCE node
W20260620 20:18:23.500990 6384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:23.500979 6387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:23.500990 6385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:23.501601 6379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:23.502130 6379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:23.503309 6379 hybrid_clock.cc:648] HybridClock initialized: now 1781986703503286 us; error 43 us; skew 500 ppm
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:23.505792 6379 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:23.506544 6379 webserver.cc:460] Webserver started at http://127.2.235.195:43755/ using document root <none> and password file <none>
I20260620 20:18:23.507027 6379 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:23.507109 6379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:23.507311 6379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:23.509101 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "5a689289888b424cbf6a317b10e81105"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "a7a1d2ceacd7ed50ea0e94b7b5d3f9f4"
server_key_iv: "6b64dd4090c38991339f3fd83acd718e"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.509696 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "5a689289888b424cbf6a317b10e81105"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "a7a1d2ceacd7ed50ea0e94b7b5d3f9f4"
server_key_iv: "6b64dd4090c38991339f3fd83acd718e"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.512985 6379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:23.515413 6394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.516691 6379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:18:23.516834 6379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "5a689289888b424cbf6a317b10e81105"
format_stamp: "Formatted at 2026-06-20 20:18:23 on dist-test-slave-8z53"
server_key: "a7a1d2ceacd7ed50ea0e94b7b5d3f9f4"
server_key_iv: "6b64dd4090c38991339f3fd83acd718e"
server_key_version: "encryptionkey@0"
I20260620 20:18:23.516923 6379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:23.540047 6379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:23.542754 6379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:23.542930 6379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:23.543443 6379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:23.544265 6379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:23.544329 6379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.544389 6379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:23.544437 6379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:23.553028 6379 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:40535
I20260620 20:18:23.553066 6507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:40535 every 8 connection(s)
I20260620 20:18:23.553836 6379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:23.556061 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6379
I20260620 20:18:23.556195 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:23.556495 2991 external_mini_cluster.cc:1428] Setting key 8d8bf8e486fdc77ac024be9d9ff9d3de
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:23.566529 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.555496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:51267 (local address 127.2.235.254:44401)
0620 20:18:23.555637 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:23.555640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:23.556471 (+ 831us) server_negotiation.cc:407] Connection header received
0620 20:18:23.557981 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:23.557984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:23.558025 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:23.558083 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:23.559285 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.559790 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.560823 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.561008 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.561075 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.563503 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:23.563505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:23.563507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:23.563528 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:23.564777 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.565241 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.565281 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.565286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.565368 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.565670 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.565672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.565674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.565821 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:23.565904 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:23.566348 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 20:18:23.566427 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:18:23.567253 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.555835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44401 (local address 127.2.235.195:51267)
0620 20:18:23.556315 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:23.556365 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:23.557755 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:23.558262 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:23.558272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:23.558704 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:23.559158 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:23.559171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.559892 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.559895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:23.560720 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:23.560729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.561132 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.561135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:23.561291 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.561939 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:23.561956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:23.563353 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:23.564882 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:23.564886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:23.564896 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:23.565149 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:23.565452 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:23.565455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:23.565456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:23.565586 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:23.565919 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:23.565923 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:23.566157 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 20:18:23.566311 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:18:23.566535 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":146,"threads_started":1}
I20260620 20:18:23.568290 6508 heartbeater.cc:347] Connected to a master server at 127.2.235.254:44401
I20260620 20:18:23.568471 6508 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:23.568856 6508 heartbeater.cc:511] Master 127.2.235.254:44401 requested a full tablet report, sending...
I20260620 20:18:23.569967 6049 ts_manager.cc:194] Registered new tserver with Master: 5a689289888b424cbf6a317b10e81105 (127.2.235.195:40535)
I20260620 20:18:23.570379 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:23.570642 6049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:51267
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986703, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:23.593439 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:23.586354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.2.235.254:44401)
0620 20:18:23.586463 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:23.586466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:23.586675 (+ 209us) server_negotiation.cc:407] Connection header received
0620 20:18:23.586775 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:23.586777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:23.586811 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:23.586876 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:23.587520 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.587970 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.588707 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:23.588884 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:23.588947 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:23.590979 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:23.590982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:23.590984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:23.591002 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:23.592083 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.592424 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.592426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.592428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.592490 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:23.592705 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:23.592707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:23.592708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:23.592886 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:23.592981 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:23.593199 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:18:23.593331 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:18:23.593784 6049 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35442
I20260620 20:18:23.594641 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6111
I20260620 20:18:23.600441 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6245
I20260620 20:18:23.608438 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6379
I20260620 20:18:23.615252 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6017
2026-06-20T20:18:23Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1925 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6520](info): set up 2 sockets
Jun 20 20:18:23 dist-test-slave-8z53 krb5kdc[6520](info): commencing operation
krb5kdc: starting...
W20260620 20:18:24.581223 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:24 dist-test-slave-8z53 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:24.685599 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40165
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:40165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:24.784008 6533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:24.784250 6533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:24.784320 6533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:24.788056 6533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:24.788128 6533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:24.788164 6533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:24.788218 6533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:24.788255 6533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:24.792822 6533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:40165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:24.793725 6533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:24.794559 6533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:24.801847 6538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:24.801854 6541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:24.801874 6539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:24.802037 6533 server_base.cc:1034] running on GCE node
I20260620 20:18:24.802496 6533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:24.803377 6533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:24.804559 6533 hybrid_clock.cc:648] HybridClock initialized: now 1781986704804544 us; error 27 us; skew 500 ppm
Jun 20 20:18:24 dist-test-slave-8z53 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986704, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:24.809706 6533 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:24.810427 6533 webserver.cc:460] Webserver started at http://127.2.235.254:43633/ using document root <none> and password file <none>
I20260620 20:18:24.810839 6533 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:24.810887 6533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:24.811022 6533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:24.812688 6533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "392df41998464536b2f9930540f9d2a6"
format_stamp: "Formatted at 2026-06-20 20:18:24 on dist-test-slave-8z53"
server_key: "8ba4eb775c2e70fd3941b9636984e436"
server_key_iv: "19ba254ae339ffd2c36edb978e52d8df"
server_key_version: "encryptionkey@0"
I20260620 20:18:24.813114 6533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "392df41998464536b2f9930540f9d2a6"
format_stamp: "Formatted at 2026-06-20 20:18:24 on dist-test-slave-8z53"
server_key: "8ba4eb775c2e70fd3941b9636984e436"
server_key_iv: "19ba254ae339ffd2c36edb978e52d8df"
server_key_version: "encryptionkey@0"
I20260620 20:18:24.816217 6533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:18:24.818269 6548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:24.819290 6533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:24.819378 6533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "392df41998464536b2f9930540f9d2a6"
format_stamp: "Formatted at 2026-06-20 20:18:24 on dist-test-slave-8z53"
server_key: "8ba4eb775c2e70fd3941b9636984e436"
server_key_iv: "19ba254ae339ffd2c36edb978e52d8df"
server_key_version: "encryptionkey@0"
I20260620 20:18:24.819456 6533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:24.849086 6533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:24.851403 6533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:24.851553 6533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:24.857820 6533 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:40165
I20260620 20:18:24.857843 6600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:40165 every 8 connection(s)
I20260620 20:18:24.858588 6533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:24.860216 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6533
I20260620 20:18:24.860301 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:24.860543 2991 external_mini_cluster.cc:1428] Setting key a18ec15d76045ad7136b934943aece1c
I20260620 20:18:24.860948 6601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:24.866104 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6: Bootstrap starting.
Jun 20 20:18:24 dist-test-slave-8z53 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:24.868111 6601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:24.868774 6601 log.cc:824] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:24.870321 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6: No bootstrap required, opened a new log
I20260620 20:18:24.872305 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:24.861680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.2.235.254:40165)
0620 20:18:24.862046 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:24.862057 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:24.862086 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:18:24.862626 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:24.862639 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:24.862894 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:24.863197 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:24.863952 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:24.864809 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:24.865698 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:24.866156 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:24.866308 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:24.868428 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:24.868432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:24.868443 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:24.868465 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:24.869932 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:24.870324 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:24.870329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:24.870335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:24.870475 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:24.870722 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:24.870726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:24.870727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:24.871059 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:24.871232 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:24.871492 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 20:18:24.871730 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260620 20:18:24.873417 6601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392df41998464536b2f9930540f9d2a6" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40165 } }
I20260620 20:18:24.873651 6601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:24.873700 6601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392df41998464536b2f9930540f9d2a6, State: Initialized, Role: FOLLOWER
I20260620 20:18:24.874147 6601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392df41998464536b2f9930540f9d2a6" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40165 } }
I20260620 20:18:24.874300 6601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:24.874358 6601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:24.874426 6601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:24.875120 6601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392df41998464536b2f9930540f9d2a6" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40165 } }
I20260620 20:18:24.875375 6601 leader_election.cc:302] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 392df41998464536b2f9930540f9d2a6; no voters:
I20260620 20:18:24.875586 6601 leader_election.cc:288] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:24.875698 6606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:24.875928 6606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 392df41998464536b2f9930540f9d2a6, State: Running, Role: LEADER
I20260620 20:18:24.876248 6606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392df41998464536b2f9930540f9d2a6" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40165 } }
I20260620 20:18:24.876701 6601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:24.877712 6608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 392df41998464536b2f9930540f9d2a6. Latest consensus state: current_term: 1 leader_uuid: "392df41998464536b2f9930540f9d2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392df41998464536b2f9930540f9d2a6" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40165 } } }
I20260620 20:18:24.877856 6608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:24.878219 6615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:24.878356 6607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "392df41998464536b2f9930540f9d2a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "392df41998464536b2f9930540f9d2a6" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40165 } } }
I20260620 20:18:24.878469 6607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:24.880971 6615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:24.886065 6615 catalog_manager.cc:1347] Generated new cluster ID: 2ba151b98ad94504967346142148228f
I20260620 20:18:24.886142 6615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:24.896448 6615 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:24.897507 6615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:24.902638 6615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 392df41998464536b2f9930540f9d2a6: Generated new TSK 0
I20260620 20:18:24.903250 6615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:24.937878 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40165
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:25.036880 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:25.037132 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:25.037207 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:25.040784 6627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:25.040851 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:25.040936 6627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:25.045555 6627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:25.046458 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:25.047313 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:25.055785 6635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:25.055791 6632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:25.055863 6633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:25.055804 6627 server_base.cc:1034] running on GCE node
I20260620 20:18:25.056330 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:25.056972 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:25.058167 6627 hybrid_clock.cc:648] HybridClock initialized: now 1781986705058130 us; error 51 us; skew 500 ppm
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:25.060530 6627 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:25.061296 6627 webserver.cc:460] Webserver started at http://127.2.235.193:42081/ using document root <none> and password file <none>
I20260620 20:18:25.061765 6627 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:25.061849 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:25.062033 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:25.063704 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "49a38e8521f1457f9a4d9d2fad1f21d6"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "53c53126e99f68d13b0701ac2668dc32"
server_key_iv: "4e1917ee7cf06a8d8a6af72c96936ba5"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.064179 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "49a38e8521f1457f9a4d9d2fad1f21d6"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "53c53126e99f68d13b0701ac2668dc32"
server_key_iv: "4e1917ee7cf06a8d8a6af72c96936ba5"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.067420 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:25.069459 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.070393 6627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:25.070523 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "49a38e8521f1457f9a4d9d2fad1f21d6"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "53c53126e99f68d13b0701ac2668dc32"
server_key_iv: "4e1917ee7cf06a8d8a6af72c96936ba5"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.070621 6627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:25.092752 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:25.095137 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:25.095320 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:25.095829 6627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:25.096553 6627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:25.096621 6627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.096706 6627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:25.096750 6627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.104897 6627 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:33161
I20260620 20:18:25.104923 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:33161 every 8 connection(s)
I20260620 20:18:25.105762 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:25.113251 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6627
I20260620 20:18:25.113416 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:25.113832 2991 external_mini_cluster.cc:1428] Setting key 79ef1b0cc3b542fb112d2b860c42f618
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:25.119369 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.107386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:44895 (local address 127.2.235.254:40165)
0620 20:18:25.107534 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:25.107539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:25.108308 (+ 769us) server_negotiation.cc:407] Connection header received
0620 20:18:25.109809 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:25.109811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:25.109855 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:25.109923 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:25.110919 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.111424 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.112451 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.112691 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.112768 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.115558 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:25.115561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:25.115563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:25.115583 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:25.117344 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.117787 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.117792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.117794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.117885 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.118212 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.118218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.118223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.118466 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:25.118573 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:25.118983 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 20:18:25.119129 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:18:25.119880 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.107753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40165 (local address 127.2.235.193:44895)
0620 20:18:25.108165 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:25.108201 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:25.109596 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:25.110082 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:25.110088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:25.110382 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:25.110797 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:25.110808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.111554 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.111556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:25.112336 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:25.112344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.112812 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.112814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:25.112937 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.113731 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:25.113754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:25.115393 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:25.117449 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:25.117454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:25.117463 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:25.117690 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:25.117976 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:25.117978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:25.117979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:25.118123 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:25.118568 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:25.118572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:25.118755 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 20:18:25.118904 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:18:25.119122 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":122,"threads_started":1}
I20260620 20:18:25.120936 6756 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40165
I20260620 20:18:25.121184 6756 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:25.121775 6756 heartbeater.cc:511] Master 127.2.235.254:40165 requested a full tablet report, sending...
I20260620 20:18:25.123447 6565 ts_manager.cc:194] Registered new tserver with Master: 49a38e8521f1457f9a4d9d2fad1f21d6 (127.2.235.193:33161)
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:25.124668 6565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:44895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:25.138152 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40165
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:25.235059 6761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:25.235258 6761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:25.235297 6761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:25.238955 6761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:25.238991 6761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:25.239037 6761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:25.243543 6761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:25.244304 6761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:25.245113 6761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:25.253226 6769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:25.253334 6761 server_base.cc:1034] running on GCE node
W20260620 20:18:25.253237 6767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:25.253222 6766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:25.253887 6761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:25.254459 6761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:25.255609 6761 hybrid_clock.cc:648] HybridClock initialized: now 1781986705255580 us; error 41 us; skew 500 ppm
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:25.258158 6761 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:25.258917 6761 webserver.cc:460] Webserver started at http://127.2.235.194:43885/ using document root <none> and password file <none>
I20260620 20:18:25.259372 6761 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:25.259456 6761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:25.259653 6761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:25.261447 6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "77688823f9c7498e87d2e57d96f3896a"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "63bc32cb8747cae9cc2acd573dae0c54"
server_key_iv: "1d21da9cb75253395e86a0fb33dc42ac"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.261979 6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "77688823f9c7498e87d2e57d96f3896a"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "63bc32cb8747cae9cc2acd573dae0c54"
server_key_iv: "1d21da9cb75253395e86a0fb33dc42ac"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.265245 6761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:25.267417 6776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.268460 6761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:25.268579 6761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "77688823f9c7498e87d2e57d96f3896a"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "63bc32cb8747cae9cc2acd573dae0c54"
server_key_iv: "1d21da9cb75253395e86a0fb33dc42ac"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.268673 6761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:25.294602 6761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:25.297534 6761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:25.297709 6761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:25.298192 6761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:25.298870 6761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:25.298938 6761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.298995 6761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:25.299039 6761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.307107 6761 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:41323
I20260620 20:18:25.307142 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:41323 every 8 connection(s)
I20260620 20:18:25.307847 6761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:25.313216 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6761
I20260620 20:18:25.313344 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:25.313621 2991 external_mini_cluster.cc:1428] Setting key 499618e1ad6de0c3e600e77d1784267e
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:25.320745 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.309401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:54739 (local address 127.2.235.254:40165)
0620 20:18:25.309521 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:25.309524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:25.310290 (+ 766us) server_negotiation.cc:407] Connection header received
0620 20:18:25.311551 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:25.311554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:25.311609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:25.311713 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:25.312700 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.313383 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.314706 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.315047 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.315148 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.317309 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:25.317314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:25.317317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:25.317347 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:25.318830 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.319345 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.319349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.319351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.319429 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.319734 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.319739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.319741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.319902 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:25.320010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:25.320472 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 20:18:25.320585 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:18:25.321527 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.309754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40165 (local address 127.2.235.194:54739)
0620 20:18:25.310152 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:25.310189 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:25.311353 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:25.311861 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:25.311867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:25.312161 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:25.312558 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:25.312569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.313504 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.313507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:25.314572 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:25.314582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.315191 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.315194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:25.315319 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.315827 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:25.315842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:25.317150 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:25.318963 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:25.318968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:25.318980 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:25.319247 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:25.319515 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:25.319520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:25.319522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:25.319648 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:25.320008 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:25.320014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:25.320257 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 20:18:25.320437 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 20:18:25.320708 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:18:25.322554 6890 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40165
I20260620 20:18:25.322785 6890 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:25.323228 6890 heartbeater.cc:511] Master 127.2.235.254:40165 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:18:25.324452 6565 ts_manager.cc:194] Registered new tserver with Master: 77688823f9c7498e87d2e57d96f3896a (127.2.235.194:41323)
I20260620 20:18:25.325009 6565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:54739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:25.337177 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40165
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:41485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:25.434176 6895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:25.434417 6895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:25.434490 6895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:25.438194 6895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:25.438277 6895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:25.438370 6895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:25.442839 6895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:25.443660 6895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:25.444543 6895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:25.452907 6895 server_base.cc:1034] running on GCE node
W20260620 20:18:25.452862 6900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:25.452880 6901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:25.452869 6903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:25.453477 6895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:25.454070 6895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:25.455241 6895 hybrid_clock.cc:648] HybridClock initialized: now 1781986705455236 us; error 34 us; skew 500 ppm
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:25.457796 6895 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:25.458505 6895 webserver.cc:460] Webserver started at http://127.2.235.195:38663/ using document root <none> and password file <none>
I20260620 20:18:25.458971 6895 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:25.459048 6895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:25.459235 6895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:25.460906 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "2fcae72e544342e9b3e6f38288c40383"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "c685b4813e54bddff3434e10f059a503"
server_key_iv: "8bc1475f74f3528b5018683127c02221"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.461467 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "2fcae72e544342e9b3e6f38288c40383"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "c685b4813e54bddff3434e10f059a503"
server_key_iv: "8bc1475f74f3528b5018683127c02221"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.464586 6895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:25.466914 6910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.468048 6895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 20:18:25.468166 6895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "2fcae72e544342e9b3e6f38288c40383"
format_stamp: "Formatted at 2026-06-20 20:18:25 on dist-test-slave-8z53"
server_key: "c685b4813e54bddff3434e10f059a503"
server_key_iv: "8bc1475f74f3528b5018683127c02221"
server_key_version: "encryptionkey@0"
I20260620 20:18:25.468259 6895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:25.497572 6895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:25.499917 6895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:25.500077 6895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:25.500551 6895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:25.501237 6895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:25.501327 6895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.501386 6895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:25.501410 6895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:25.509384 6895 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:33841
I20260620 20:18:25.509425 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:33841 every 8 connection(s)
I20260620 20:18:25.510133 6895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:25.512013 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 6895
I20260620 20:18:25.512122 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:25.512358 2991 external_mini_cluster.cc:1428] Setting key ecaf9eab147e97f5d969643ada738f29
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:25.521926 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.511653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:57623 (local address 127.2.235.254:40165)
0620 20:18:25.511769 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:25.511773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:25.512440 (+ 667us) server_negotiation.cc:407] Connection header received
0620 20:18:25.513901 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:25.513904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:25.513950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:25.514026 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:25.514977 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.515489 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.516515 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.516724 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.516794 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.518882 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:25.518884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:25.518886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:25.518908 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:25.520225 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.520652 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.520655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.520656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.520722 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.521030 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.521032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.521033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.521185 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:25.521295 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:25.521681 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 20:18:25.521816 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:18:25.522629 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.511938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40165 (local address 127.2.235.195:57623)
0620 20:18:25.512319 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:25.512361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:25.513736 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:25.514185 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:25.514197 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:25.514476 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:25.514841 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:25.514850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.515596 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.515598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:25.516423 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:25.516431 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.516842 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.516844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:25.516981 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.517470 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:25.517485 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:25.518733 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:25.520333 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:25.520337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:25.520346 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:25.520572 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:25.520832 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:25.520834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:25.520836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:25.520958 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:25.521319 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:25.521324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:25.521505 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 20:18:25.521639 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:18:25.521920 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":121,"threads_started":1}
I20260620 20:18:25.523630 7024 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40165
I20260620 20:18:25.523797 7024 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:25.524153 7024 heartbeater.cc:511] Master 127.2.235.254:40165 requested a full tablet report, sending...
I20260620 20:18:25.525099 6564 ts_manager.cc:194] Registered new tserver with Master: 2fcae72e544342e9b3e6f38288c40383 (127.2.235.195:33841)
I20260620 20:18:25.525820 6564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:57623
I20260620 20:18:25.526217 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:25.549583 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.540960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.2.235.193:33161)
0620 20:18:25.541263 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:25.541277 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:25.541300 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:18:25.541354 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:25.541358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:25.541509 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:25.541637 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:25.542287 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.543020 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.543822 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.544137 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.544244 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.546469 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:25.546476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:25.546486 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:25.546506 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:25.547923 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.548325 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.548334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.548335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.548411 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.548668 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.548672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.548674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.548872 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:25.549061 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:25.549299 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:18:25.549440 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:25.558904 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:25.552023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.2.235.254:40165)
0620 20:18:25.552152 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:25.552155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:25.552168 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:25.552249 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:25.552251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:25.552282 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:25.552340 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:25.552854 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.553403 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.554068 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:25.554241 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:25.554304 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:25.556391 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:25.556393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:25.556395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:25.556416 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:25.557597 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.557908 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.557910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.557911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.557989 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:25.558184 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:25.558186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:25.558187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:25.558348 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:25.558442 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:25.558654 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 20:18:25.558729 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:18:25.559243 6564 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52628
I20260620 20:18:25.559711 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6627
I20260620 20:18:25.565160 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6761
I20260620 20:18:25.570313 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6895
I20260620 20:18:25.575042 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 6533
2026-06-20T20:18:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1956 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[7036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[7036](info): set up 2 sockets
Jun 20 20:18:25 dist-test-slave-8z53 krb5kdc[7036](info): commencing operation
krb5kdc: starting...
W20260620 20:18:26.541214 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:26 dist-test-slave-8z53 krb5kdc[7036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:26.646092 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:39731
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:26.746212 7049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:26.746511 7049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:26.746582 7049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:26.750281 7049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:26.750350 7049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:26.750383 7049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:26.750419 7049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:26.750468 7049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:26.754989 7049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:39731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:26.755879 7049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:26.756770 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:26.764055 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:26.764093 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:26.764065 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:26.764155 7049 server_base.cc:1034] running on GCE node
I20260620 20:18:26.764904 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:26.766230 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:26.767433 7049 hybrid_clock.cc:648] HybridClock initialized: now 1781986706767403 us; error 40 us; skew 500 ppm
Jun 20 20:18:26 dist-test-slave-8z53 krb5kdc[7036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986706, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:26.770174 7049 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:26.770808 7049 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:26.770874 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:26.771090 7049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:26.772816 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "71252084b6c249b79c09756fbd9efb33"
format_stamp: "Formatted at 2026-06-20 20:18:26 on dist-test-slave-8z53"
server_key: "0b7cb916dd802af30918637ed1b30138"
server_key_iv: "52290267e6b0805bb58be5e9d81b5970"
server_key_version: "encryptionkey@0"
I20260620 20:18:26.773417 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "71252084b6c249b79c09756fbd9efb33"
format_stamp: "Formatted at 2026-06-20 20:18:26 on dist-test-slave-8z53"
server_key: "0b7cb916dd802af30918637ed1b30138"
server_key_iv: "52290267e6b0805bb58be5e9d81b5970"
server_key_version: "encryptionkey@0"
I20260620 20:18:26.776865 7049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:26.779125 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:26.780225 7049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:18:26.780375 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "71252084b6c249b79c09756fbd9efb33"
format_stamp: "Formatted at 2026-06-20 20:18:26 on dist-test-slave-8z53"
server_key: "0b7cb916dd802af30918637ed1b30138"
server_key_iv: "52290267e6b0805bb58be5e9d81b5970"
server_key_version: "encryptionkey@0"
I20260620 20:18:26.780483 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:26.793195 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:26.796242 7049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:26.796422 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:26.803061 7049 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:39731
I20260620 20:18:26.803283 7115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:39731 every 8 connection(s)
I20260620 20:18:26.803660 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:26.806030 7116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:26.810659 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7049
I20260620 20:18:26.810777 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:26.810957 7116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33: Bootstrap starting.
I20260620 20:18:26.811064 2991 external_mini_cluster.cc:1428] Setting key 2156933cf7aa00d923324954fb992b12
I20260620 20:18:26.813185 7116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:26.813982 7116 log.cc:824] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:26.815721 7116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33: No bootstrap required, opened a new log
Jun 20 20:18:26 dist-test-slave-8z53 krb5kdc[7036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:26.819625 7116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71252084b6c249b79c09756fbd9efb33" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39731 } }
I20260620 20:18:26.819825 7116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:26.819916 7116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71252084b6c249b79c09756fbd9efb33, State: Initialized, Role: FOLLOWER
I20260620 20:18:26.820322 7116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71252084b6c249b79c09756fbd9efb33" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39731 } }
I20260620 20:18:26.820463 7116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:26.820560 7116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:26.820655 7116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:26.821750 7116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71252084b6c249b79c09756fbd9efb33" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39731 } }
I20260620 20:18:26.822095 7116 leader_election.cc:302] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71252084b6c249b79c09756fbd9efb33; no voters:
I20260620 20:18:26.822432 7116 leader_election.cc:288] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:26.822479 7121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:26.822657 7121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [term 1 LEADER]: Becoming Leader. State: Replica: 71252084b6c249b79c09756fbd9efb33, State: Running, Role: LEADER
I20260620 20:18:26.822930 7121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71252084b6c249b79c09756fbd9efb33" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39731 } }
I20260620 20:18:26.823521 7116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:26.823973 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71252084b6c249b79c09756fbd9efb33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71252084b6c249b79c09756fbd9efb33" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39731 } } }
I20260620 20:18:26.824095 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:26.824366 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71252084b6c249b79c09756fbd9efb33. Latest consensus state: current_term: 1 leader_uuid: "71252084b6c249b79c09756fbd9efb33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71252084b6c249b79c09756fbd9efb33" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39731 } } }
I20260620 20:18:26.824465 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:26.825479 7127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:26.825745 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:26.812185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.2.235.254:39731)
0620 20:18:26.812523 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:26.812539 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:26.812580 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:18:26.813163 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:26.813177 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:26.813552 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:26.813795 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:26.814621 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:26.815595 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:26.816713 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:26.817158 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:26.817315 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:26.819931 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:26.819936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:26.819947 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:26.819973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:26.822067 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:26.822571 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:26.822578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:26.822583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:26.822771 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:26.823097 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:26.823101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:26.823103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:26.823527 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:26.823716 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:26.824549 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 20:18:26.824808 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":127,"threads_started":1}
I20260620 20:18:26.828173 7127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:26.833050 7127 catalog_manager.cc:1347] Generated new cluster ID: b2f7eeada7354e31bbab84ec3fec8584
I20260620 20:18:26.833112 7127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:26.841785 7127 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:26.842948 7127 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:26.851667 7127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33: Generated new TSK 0
I20260620 20:18:26.852201 7127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:26.877738 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39731
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:27.005713 7142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:27.005931 7142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:27.005975 7142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:27.009727 7142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:27.009784 7142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:27.009840 7142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:27.014547 7142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:27.015349 7142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:27.016204 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:27.024924 7142 server_base.cc:1034] running on GCE node
W20260620 20:18:27.024855 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:27.024855 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:27.024889 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:27.025592 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:27.026176 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:27.027413 7142 hybrid_clock.cc:648] HybridClock initialized: now 1781986707027398 us; error 33 us; skew 500 ppm
Jun 20 20:18:27 dist-test-slave-8z53 krb5kdc[7036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:27.030175 7142 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:27.030854 7142 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:27.030939 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:27.031145 7142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:27.032896 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "9dc07b24f1de4530a7c7c4d8ebb11219"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "c0e4d79322ac3e1c84ac68a84fd619a5"
server_key_iv: "22fbff74d161b47f46090e57e2925d55"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.033449 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "9dc07b24f1de4530a7c7c4d8ebb11219"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "c0e4d79322ac3e1c84ac68a84fd619a5"
server_key_iv: "22fbff74d161b47f46090e57e2925d55"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.036736 7142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:18:27.038846 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.039956 7142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:27.040083 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "9dc07b24f1de4530a7c7c4d8ebb11219"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "c0e4d79322ac3e1c84ac68a84fd619a5"
server_key_iv: "22fbff74d161b47f46090e57e2925d55"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.040179 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:27.051178 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:27.054015 7142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:27.054185 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:27.054666 7142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:27.055402 7142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:27.055469 7142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.055536 7142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:27.055580 7142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.063966 7142 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:33597
I20260620 20:18:27.063997 7269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:33597 every 8 connection(s)
I20260620 20:18:27.064457 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
Jun 20 20:18:27 dist-test-slave-8z53 krb5kdc[7036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:27.073096 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7142
I20260620 20:18:27.073208 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:27.073475 2991 external_mini_cluster.cc:1428] Setting key eacefdb908861436ae86428265fc338f
I20260620 20:18:27.076588 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.065971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:40677 (local address 127.2.235.254:39731)
0620 20:18:27.066098 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.066101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.066808 (+ 707us) server_negotiation.cc:407] Connection header received
0620 20:18:27.067999 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.068002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.068055 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.068146 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:27.069108 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.069670 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.070684 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.070909 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.071010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.072972 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:27.072975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:27.072977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:27.072997 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:27.074592 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.075077 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.075082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.075083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.075177 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.075490 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.075496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.075501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.075741 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:27.075840 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.076326 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.076451 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:18:27.077586 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.066248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:39731 (local address 127.2.235.193:40677)
0620 20:18:27.066660 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.066697 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.067789 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.068293 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.068300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.068583 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:27.068970 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.068981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.069800 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.069803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.070558 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.070566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.071041 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.071044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.071190 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.071586 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:27.071600 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:27.072799 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:27.074685 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:27.074690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:27.074703 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:27.074961 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:27.075276 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:27.075278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:27.075279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:27.075398 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:27.075842 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:27.075847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:27.076051 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.076249 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.076586 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260620 20:18:27.078635 7270 heartbeater.cc:347] Connected to a master server at 127.2.235.254:39731
I20260620 20:18:27.078841 7270 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:27.079219 7270 heartbeater.cc:511] Master 127.2.235.254:39731 requested a full tablet report, sending...
I20260620 20:18:27.080497 7080 ts_manager.cc:194] Registered new tserver with Master: 9dc07b24f1de4530a7c7c4d8ebb11219 (127.2.235.193:33597)
I20260620 20:18:27.081588 7080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:40677
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:27.097177 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39731
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:27.191645 7275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:27.191851 7275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:27.191910 7275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:27.195463 7275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:27.195537 7275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:27.195623 7275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:27.199898 7275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:27.200687 7275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:27.201566 7275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:27.209678 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:27.209636 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:27.209633 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:27.209666 7275 server_base.cc:1034] running on GCE node
I20260620 20:18:27.210163 7275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:27.210733 7275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:27.211907 7275 hybrid_clock.cc:648] HybridClock initialized: now 1781986707211893 us; error 24 us; skew 500 ppm
Jun 20 20:18:27 dist-test-slave-8z53 krb5kdc[7036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:27.214490 7275 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:27.215090 7275 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:27.215149 7275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:27.215287 7275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:27.217005 7275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "ed1f61609c9c43ec93920e04452f10ae"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "7c3be40a79e90744cc88f786319fc581"
server_key_iv: "3b8b66867a19c92f9a1527ab4766061e"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.217525 7275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "ed1f61609c9c43ec93920e04452f10ae"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "7c3be40a79e90744cc88f786319fc581"
server_key_iv: "3b8b66867a19c92f9a1527ab4766061e"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.220444 7275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:18:27.222554 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.223582 7275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:27.223656 7275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "ed1f61609c9c43ec93920e04452f10ae"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "7c3be40a79e90744cc88f786319fc581"
server_key_iv: "3b8b66867a19c92f9a1527ab4766061e"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.223718 7275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:27.240959 7275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:27.243743 7275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:27.243849 7275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:27.244275 7275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:27.244866 7275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:27.244923 7275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.244956 7275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:27.244969 7275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.253067 7275 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:36379
I20260620 20:18:27.253085 7402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:36379 every 8 connection(s)
I20260620 20:18:27.253552 7275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:27.261605 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7275
Jun 20 20:18:27 dist-test-slave-8z53 krb5kdc[7036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:27.261750 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:27.262037 2991 external_mini_cluster.cc:1428] Setting key 5611ce2053c32d6ee6a2ddac1bb5efab
I20260620 20:18:27.265915 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.255144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:36109 (local address 127.2.235.254:39731)
0620 20:18:27.255283 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.255287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.255993 (+ 706us) server_negotiation.cc:407] Connection header received
0620 20:18:27.257166 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.257169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.257210 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.257326 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:27.258346 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.258850 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.259917 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.260111 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.260181 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.262352 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:27.262359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:27.262364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:27.262391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:27.263930 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.264362 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.264368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.264372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.264459 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.264752 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.264757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.264762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.264998 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:27.265127 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.265695 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.265785 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:18:27.266794 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.255435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:39731 (local address 127.2.235.194:36109)
0620 20:18:27.255838 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.255883 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.256992 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.257468 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.257473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.257790 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:27.258229 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.258241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.258969 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.258972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.259789 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.259800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.260224 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.260227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.260344 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.260730 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:27.260743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:27.262203 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:27.264033 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:27.264037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:27.264046 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:27.264271 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:27.264554 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:27.264556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:27.264558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:27.264668 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:27.265138 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:27.265145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:27.265449 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.265642 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.265956 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":107,"threads_started":1}
I20260620 20:18:27.267956 7403 heartbeater.cc:347] Connected to a master server at 127.2.235.254:39731
I20260620 20:18:27.268153 7403 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:27.268513 7403 heartbeater.cc:511] Master 127.2.235.254:39731 requested a full tablet report, sending...
I20260620 20:18:27.269441 7080 ts_manager.cc:194] Registered new tserver with Master: ed1f61609c9c43ec93920e04452f10ae (127.2.235.194:36379)
I20260620 20:18:27.269958 7080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:36109
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:27.285880 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39731
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:27.384768 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:27.385001 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:27.385092 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:27.388736 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:27.388780 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:27.388865 7408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:27.393644 7408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:27.394376 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:27.395181 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:27.403327 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:27.403385 7408 server_base.cc:1034] running on GCE node
W20260620 20:18:27.403393 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:27.403532 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:27.403923 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:27.404452 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:27.405619 7408 hybrid_clock.cc:648] HybridClock initialized: now 1781986707405593 us; error 38 us; skew 500 ppm
Jun 20 20:18:27 dist-test-slave-8z53 krb5kdc[7036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:27.408116 7408 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:27.408687 7408 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:27.408776 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:27.408965 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:27.410614 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "02ce8e180ed34910ba2475a338e25e2f"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "70c1087194333bd8eac11a78903793dc"
server_key_iv: "ede878193ef6097bad23112999de66d1"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.411113 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "02ce8e180ed34910ba2475a338e25e2f"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "70c1087194333bd8eac11a78903793dc"
server_key_iv: "ede878193ef6097bad23112999de66d1"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.414114 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:18:27.416406 7422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.417608 7408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:27.417714 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "02ce8e180ed34910ba2475a338e25e2f"
format_stamp: "Formatted at 2026-06-20 20:18:27 on dist-test-slave-8z53"
server_key: "70c1087194333bd8eac11a78903793dc"
server_key_iv: "ede878193ef6097bad23112999de66d1"
server_key_version: "encryptionkey@0"
I20260620 20:18:27.417814 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:27.445823 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:27.448343 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:27.448493 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:27.448931 7408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:27.449693 7408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:27.449733 7408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.449765 7408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:27.449779 7408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:27.457180 7408 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:46087
I20260620 20:18:27.457263 7535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:46087 every 8 connection(s)
I20260620 20:18:27.457728 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:27.460487 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7408
I20260620 20:18:27.460573 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:27.460845 2991 external_mini_cluster.cc:1428] Setting key 5aeb225bbe1911f2c0eb3052ba1db9f6
Jun 20 20:18:27 dist-test-slave-8z53 krb5kdc[7036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986707, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:27.469975 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.459145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:58633 (local address 127.2.235.254:39731)
0620 20:18:27.459268 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.459272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.460064 (+ 792us) server_negotiation.cc:407] Connection header received
0620 20:18:27.461486 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.461489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.461538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.461641 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:27.462898 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.463374 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.464323 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.464532 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.464598 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.466853 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:27.466856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:27.466858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:27.466879 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:27.468198 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.468664 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.468667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.468668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.468736 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.469058 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.469061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.469062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.469225 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:27.469369 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.469773 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.469851 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:27.470686 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.459499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:39731 (local address 127.2.235.195:58633)
0620 20:18:27.459917 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.459953 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.461298 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.461804 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.461810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.462252 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:27.462777 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.462790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.463508 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.463511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.464234 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.464242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.464636 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.464638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.464762 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.465338 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:27.465352 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:27.466707 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:27.468303 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:27.468308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:27.468317 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:27.468573 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:27.468820 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:27.468822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:27.468823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:27.468963 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:27.469390 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:27.469395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:27.469581 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.469735 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.469978 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260620 20:18:27.471699 7536 heartbeater.cc:347] Connected to a master server at 127.2.235.254:39731
I20260620 20:18:27.471909 7536 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:27.472278 7536 heartbeater.cc:511] Master 127.2.235.254:39731 requested a full tablet report, sending...
I20260620 20:18:27.473235 7080 ts_manager.cc:194] Registered new tserver with Master: 02ce8e180ed34910ba2475a338e25e2f (127.2.235.195:46087)
I20260620 20:18:27.473865 7080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:58633
I20260620 20:18:27.474186 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:27.481923 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.476120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.2.235.254:39731)
0620 20:18:27.476219 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.476222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.476443 (+ 221us) server_negotiation.cc:407] Connection header received
0620 20:18:27.476529 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.476530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.476557 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.476620 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:27.477217 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.477670 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.478415 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.478591 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.478657 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.479751 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:27.479753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:27.479754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:27.479767 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:27.480702 (+ 935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.481018 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.481020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.481021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.481077 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:27.481327 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:27.481329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:27.481330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:27.481517 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:27.481616 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.481764 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.481831 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:27.484370 7080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:27.487115 7080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:27.499431 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.494596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:33597 (local address 127.0.0.1:45154)
0620 20:18:27.495043 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.495055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.495163 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.495917 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.495920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.495939 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:27.496010 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.496016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.497270 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.497275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.498068 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.498075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.499020 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.499024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.499130 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.499147 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.499211 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.499323 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":43,"thread_start_us":72,"threads_started":1}
I20260620 20:18:27.499507 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.494722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.2.235.195:46087)
0620 20:18:27.495005 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.495014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.495031 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:18:27.495101 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.495106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.495302 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.495761 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:27.495910 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.496702 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.498442 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.498870 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.498974 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.499084 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.499298 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.499369 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
I20260620 20:18:27.499716 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.494132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.2.235.194:36379)
0620 20:18:27.494922 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.494933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.494957 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:18:27.495028 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.495033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.495285 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.495436 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:27.495848 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.496949 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.498252 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.498889 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.498990 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.499110 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.499489 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.499597 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":110,"threads_started":1}
I20260620 20:18:27.499745 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.494757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.2.235.193:33597)
0620 20:18:27.495466 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.495480 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.495501 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:18:27.495576 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.495582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.495747 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.495913 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:27.496138 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.497137 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.498182 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.498892 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.499142 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.499248 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.499292 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.499369 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":142,"threads_started":1}
I20260620 20:18:27.499874 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.494404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:46087 (local address 127.0.0.1:42862)
0620 20:18:27.494647 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.494659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.494727 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.495476 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.495482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.495502 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:27.495621 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.495630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.496810 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.496815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.498336 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.498345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.498984 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.498987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.499076 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.499129 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.499214 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.499734 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":674,"thread_start_us":86,"threads_started":1}
I20260620 20:18:27.500545 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.494023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:36379 (local address 127.0.0.1:55854)
0620 20:18:27.494290 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.494331 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.494434 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.495602 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.495605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.495622 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:27.495713 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.495719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.497106 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.497109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.498148 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.498156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.499159 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.499162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.499228 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.499246 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.499308 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.499379 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":94,"thread_start_us":56,"threads_started":1}
I20260620 20:18:27.502175 7204 tablet_service.cc:1467] Processing CreateTablet for tablet eb7b001d26c94d248695bf693dd1cfdb (DEFAULT_TABLE table=test-table [id=9450e0ac59214ec2aa1cea72e7412de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:27.502197 7470 tablet_service.cc:1467] Processing CreateTablet for tablet eb7b001d26c94d248695bf693dd1cfdb (DEFAULT_TABLE table=test-table [id=9450e0ac59214ec2aa1cea72e7412de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:27.503028 7204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7b001d26c94d248695bf693dd1cfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:27.503080 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7b001d26c94d248695bf693dd1cfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:27.503343 7337 tablet_service.cc:1467] Processing CreateTablet for tablet eb7b001d26c94d248695bf693dd1cfdb (DEFAULT_TABLE table=test-table [id=9450e0ac59214ec2aa1cea72e7412de7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:27.504136 7337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7b001d26c94d248695bf693dd1cfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:27.508572 7553 tablet_bootstrap.cc:492] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: Bootstrap starting.
I20260620 20:18:27.509728 7554 tablet_bootstrap.cc:492] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Bootstrap starting.
I20260620 20:18:27.511276 7553 tablet_bootstrap.cc:654] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:27.511577 7554 tablet_bootstrap.cc:654] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:27.512539 7553 log.cc:824] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:27.512594 7554 log.cc:824] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:27.513088 7555 tablet_bootstrap.cc:492] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Bootstrap starting.
I20260620 20:18:27.514144 7553 tablet_bootstrap.cc:492] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: No bootstrap required, opened a new log
I20260620 20:18:27.514344 7553 ts_tablet_manager.cc:1397] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:18:27.514650 7554 tablet_bootstrap.cc:492] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: No bootstrap required, opened a new log
I20260620 20:18:27.514815 7554 ts_tablet_manager.cc:1397] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:27.515102 7555 tablet_bootstrap.cc:654] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:27.516548 7555 log.cc:824] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:27.517616 7553 raft_consensus.cc:348] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.517812 7553 raft_consensus.cc:374] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:27.517858 7553 raft_consensus.cc:729] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc07b24f1de4530a7c7c4d8ebb11219, State: Initialized, Role: FOLLOWER
I20260620 20:18:27.518200 7555 tablet_bootstrap.cc:492] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: No bootstrap required, opened a new log
I20260620 20:18:27.518217 7553 consensus_queue.cc:260] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.518359 7555 ts_tablet_manager.cc:1397] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:18:27.518263 7554 raft_consensus.cc:348] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.518489 7554 raft_consensus.cc:374] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:27.518543 7554 raft_consensus.cc:729] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ce8e180ed34910ba2475a338e25e2f, State: Initialized, Role: FOLLOWER
I20260620 20:18:27.518879 7554 consensus_queue.cc:260] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.519047 7553 ts_tablet_manager.cc:1428] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:18:27.519342 7270 heartbeater.cc:503] Master 127.2.235.254:39731 was elected leader, sending a full tablet report...
I20260620 20:18:27.519649 7554 ts_tablet_manager.cc:1428] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:18:27.519928 7536 heartbeater.cc:503] Master 127.2.235.254:39731 was elected leader, sending a full tablet report...
I20260620 20:18:27.521467 7555 raft_consensus.cc:348] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.521665 7555 raft_consensus.cc:374] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:27.521714 7555 raft_consensus.cc:729] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1f61609c9c43ec93920e04452f10ae, State: Initialized, Role: FOLLOWER
I20260620 20:18:27.522095 7555 consensus_queue.cc:260] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.522683 7403 heartbeater.cc:503] Master 127.2.235.254:39731 was elected leader, sending a full tablet report...
I20260620 20:18:27.522935 7555 ts_tablet_manager.cc:1428] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 20:18:27.565382 7271 tablet.cc:2369] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:27.708513 7537 tablet.cc:2369] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:27.754618 7404 tablet.cc:2369] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:27.841418 7561 raft_consensus.cc:482] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:27.841629 7561 raft_consensus.cc:504] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.842669 7561 leader_election.cc:288] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dc07b24f1de4530a7c7c4d8ebb11219 (127.2.235.193:33597), 02ce8e180ed34910ba2475a338e25e2f (127.2.235.195:46087)
I20260620 20:18:27.847714 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.842974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:54683 (local address 127.2.235.193:33597)
0620 20:18:27.843095 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.843100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.843258 (+ 158us) server_negotiation.cc:407] Connection header received
0620 20:18:27.843397 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.843402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.843450 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.843527 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:27.843933 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.844639 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.846437 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.846991 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.847102 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.847149 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.847520 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.847592 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:18:27.847714 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.843036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:39615 (local address 127.2.235.195:46087)
0620 20:18:27.843183 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.843187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.843346 (+ 159us) server_negotiation.cc:407] Connection header received
0620 20:18:27.843435 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.843437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.843484 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.843546 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:27.843904 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.844639 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.846437 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.846980 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.847102 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.847149 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.847520 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.847592 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:18:27.847787 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.842906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:46087 (local address 127.2.235.194:39615)
0620 20:18:27.843264 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.843275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.843345 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.843563 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.843566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.843602 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:27.843796 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.843803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.844742 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.844745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.846310 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.846320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.847109 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.847113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.847291 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.847407 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.847484 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.847599 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":335,"spinlock_wait_cycles":6272,"thread_start_us":55,"threads_started":1}
I20260620 20:18:27.847786 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.842840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:33597 (local address 127.2.235.194:54683)
0620 20:18:27.843169 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.843182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:27.843288 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:27.843563 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:27.843566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:27.843601 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:27.843800 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.843804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.844742 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.844745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.846310 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:27.846320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.847109 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.847113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:27.847293 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.847406 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 20:18:27.847484 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:18:27.847599 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":206,"spinlock_wait_cycles":55296,"thread_start_us":93,"threads_started":1}
I20260620 20:18:27.848471 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7b001d26c94d248695bf693dd1cfdb" candidate_uuid: "ed1f61609c9c43ec93920e04452f10ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" is_pre_election: true
I20260620 20:18:27.848471 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7b001d26c94d248695bf693dd1cfdb" candidate_uuid: "ed1f61609c9c43ec93920e04452f10ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ce8e180ed34910ba2475a338e25e2f" is_pre_election: true
I20260620 20:18:27.848765 7224 raft_consensus.cc:2454] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1f61609c9c43ec93920e04452f10ae in term 0.
I20260620 20:18:27.848765 7490 raft_consensus.cc:2454] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1f61609c9c43ec93920e04452f10ae in term 0.
I20260620 20:18:27.849170 7291 leader_election.cc:302] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dc07b24f1de4530a7c7c4d8ebb11219, ed1f61609c9c43ec93920e04452f10ae; no voters:
I20260620 20:18:27.849435 7561 raft_consensus.cc:2781] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:27.849530 7561 raft_consensus.cc:482] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:27.849570 7561 raft_consensus.cc:3037] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:27.850566 7561 raft_consensus.cc:504] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.850864 7561 leader_election.cc:288] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [CANDIDATE]: Term 1 election: Requested vote from peers 9dc07b24f1de4530a7c7c4d8ebb11219 (127.2.235.193:33597), 02ce8e180ed34910ba2475a338e25e2f (127.2.235.195:46087)
I20260620 20:18:27.851156 7224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7b001d26c94d248695bf693dd1cfdb" candidate_uuid: "ed1f61609c9c43ec93920e04452f10ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219"
I20260620 20:18:27.851271 7224 raft_consensus.cc:3037] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:27.852131 7224 raft_consensus.cc:2454] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1f61609c9c43ec93920e04452f10ae in term 1.
I20260620 20:18:27.852322 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7b001d26c94d248695bf693dd1cfdb" candidate_uuid: "ed1f61609c9c43ec93920e04452f10ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ce8e180ed34910ba2475a338e25e2f"
I20260620 20:18:27.852356 7291 leader_election.cc:302] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dc07b24f1de4530a7c7c4d8ebb11219, ed1f61609c9c43ec93920e04452f10ae; no voters:
I20260620 20:18:27.852416 7490 raft_consensus.cc:3037] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:27.852538 7561 raft_consensus.cc:2781] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:27.852720 7561 raft_consensus.cc:686] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 1 LEADER]: Becoming Leader. State: Replica: ed1f61609c9c43ec93920e04452f10ae, State: Running, Role: LEADER
I20260620 20:18:27.853025 7561 consensus_queue.cc:237] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } }
I20260620 20:18:27.853677 7490 raft_consensus.cc:2454] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1f61609c9c43ec93920e04452f10ae in term 1.
I20260620 20:18:27.856073 7078 catalog_manager.cc:5495] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1f61609c9c43ec93920e04452f10ae (127.2.235.194). New cstate: current_term: 1 leader_uuid: "ed1f61609c9c43ec93920e04452f10ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1f61609c9c43ec93920e04452f10ae" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:27.944943 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.941615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.2.235.194:36379)
0620 20:18:27.941743 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:27.941747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:27.941762 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:18:27.941882 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:27.941885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:27.941933 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:27.942025 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:27.942255 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.942717 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.943561 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:27.943777 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:27.943880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:27.944156 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:27.944605 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:27.944727 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:27.944764 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 20:18:27.944812 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:18:27.950284 7490 raft_consensus.cc:1261] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 1 FOLLOWER]: Refusing update from remote peer ed1f61609c9c43ec93920e04452f10ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:27.950332 7224 raft_consensus.cc:1261] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 1 FOLLOWER]: Refusing update from remote peer ed1f61609c9c43ec93920e04452f10ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:27.950980 7561 consensus_queue.cc:1035] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ce8e180ed34910ba2475a338e25e2f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:27.951171 7564 consensus_queue.cc:1035] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dc07b24f1de4530a7c7c4d8ebb11219" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:27.959545 7568 mvcc.cc:205] Tried to move back new op lower bound from 7299017555758039040 to 7299017555369037824. Current Snapshot: MvccSnapshot[applied={T|T < 7299017555758039040}]
I20260620 20:18:27.966970 7079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:27.967119 7079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:27.969238 7079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71252084b6c249b79c09756fbd9efb33: Sending DeleteTablet for 3 replicas of tablet eb7b001d26c94d248695bf693dd1cfdb
I20260620 20:18:27.969977 7204 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7b001d26c94d248695bf693dd1cfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:27 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:45154
I20260620 20:18:27.970023 7470 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7b001d26c94d248695bf693dd1cfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:27 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:42862
I20260620 20:18:27.970106 7337 tablet_service.cc:1514] Processing DeleteTablet for tablet eb7b001d26c94d248695bf693dd1cfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:27 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:55854
I20260620 20:18:27.970391 7577 tablet_replica.cc:330] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219: stopping tablet replica
I20260620 20:18:27.970549 7578 tablet_replica.cc:330] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: stopping tablet replica
I20260620 20:18:27.970669 7577 raft_consensus.cc:2229] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:27.970870 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7142
I20260620 20:18:27.970955 7577 raft_consensus.cc:2258] T eb7b001d26c94d248695bf693dd1cfdb P 9dc07b24f1de4530a7c7c4d8ebb11219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:27.971933 7578 raft_consensus.cc:2229] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:27.972190 7578 raft_consensus.cc:2258] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:27.973299 7578 ts_tablet_manager.cc:1905] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:27.973603 7579 tablet_replica.cc:330] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: stopping tablet replica
I20260620 20:18:27.973834 7579 raft_consensus.cc:2229] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:27.974139 7579 raft_consensus.cc:2258] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:27.975391 7579 ts_tablet_manager.cc:1905] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:27.975818 7578 ts_tablet_manager.cc:1918] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:27.975924 7578 log.cc:1196] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/eb7b001d26c94d248695bf693dd1cfdb
I20260620 20:18:27.976222 7578 ts_tablet_manager.cc:1939] T eb7b001d26c94d248695bf693dd1cfdb P 02ce8e180ed34910ba2475a338e25e2f: Deleting consensus metadata
I20260620 20:18:27.977301 7067 catalog_manager.cc:4841] TS 02ce8e180ed34910ba2475a338e25e2f (127.2.235.195:46087): tablet eb7b001d26c94d248695bf693dd1cfdb (table test-table [id=9450e0ac59214ec2aa1cea72e7412de7]) successfully deleted
W20260620 20:18:27.978456 7065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:33597 (error 108)
I20260620 20:18:27.978590 7579 ts_tablet_manager.cc:1918] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:27.978677 7579 log.cc:1196] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/eb7b001d26c94d248695bf693dd1cfdb
I20260620 20:18:27.978960 7579 ts_tablet_manager.cc:1939] T eb7b001d26c94d248695bf693dd1cfdb P ed1f61609c9c43ec93920e04452f10ae: Deleting consensus metadata
I20260620 20:18:27.979156 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7275
I20260620 20:18:27.980346 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.980020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:33597 (local address 127.0.0.1:45160)
0620 20:18:27.980161 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.980234 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:33597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:18:27.980545 7065 catalog_manager.cc:4584] TS 9dc07b24f1de4530a7c7c4d8ebb11219 (127.2.235.193:33597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb7b001d26c94d248695bf693dd1cfdb: Network error: Client connection negotiation failed: client connection to 127.2.235.193:33597: connect: Connection refused (error 111)
I20260620 20:18:27.984692 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7408
I20260620 20:18:27.986469 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:27.984823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:36379 (local address 127.0.0.1:55860)
0620 20:18:27.986296 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:27.986388 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.194:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1381}
I20260620 20:18:27.990113 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7049
2026-06-20T20:18:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2414 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T20:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:28Z Disabled control of system clock
I20260620 20:18:28.017889 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42141
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:18:28.113999 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:28.114174 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:28.114202 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:28.117722 7584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:28.117774 7584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:28.117787 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:28.117801 7584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:28.117815 7584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:28.121805 7584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:42141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:28.122500 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:28.123466 7584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:28.130745 7592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:28.130749 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:28.130785 7584 server_base.cc:1034] running on GCE node
W20260620 20:18:28.130734 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:28.131417 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:28.132282 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:28.133422 7584 hybrid_clock.cc:648] HybridClock initialized: now 1781986708133393 us; error 42 us; skew 500 ppm
I20260620 20:18:28.134887 7584 webserver.cc:460] Webserver started at http://127.2.235.254:39747/ using document root <none> and password file <none>
I20260620 20:18:28.135341 7584 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:28.135393 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:28.135529 7584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:28.137217 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "f422a2858bdb4a1bbb5b32acdab39f9f"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "1fccc3ad62c3dfda59a376ae5ec23e68"
server_key_iv: "71588fba1cca929c6021bd8a9a6beb30"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.137697 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "f422a2858bdb4a1bbb5b32acdab39f9f"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "1fccc3ad62c3dfda59a376ae5ec23e68"
server_key_iv: "71588fba1cca929c6021bd8a9a6beb30"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.140799 7584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:28.142755 7598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.143787 7584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:28.143880 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "f422a2858bdb4a1bbb5b32acdab39f9f"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "1fccc3ad62c3dfda59a376ae5ec23e68"
server_key_iv: "71588fba1cca929c6021bd8a9a6beb30"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.143949 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:28.147801 7584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:28.148242 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:28.148353 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:28.154491 7584 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:42141
I20260620 20:18:28.154500 7650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:42141 every 8 connection(s)
I20260620 20:18:28.155273 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:28.158015 7651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:28.162257 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7584
I20260620 20:18:28.162371 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:28.162644 2991 external_mini_cluster.cc:1428] Setting key 35e6e98748e9f5f073895c8474e81442
I20260620 20:18:28.162644 7651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f: Bootstrap starting.
I20260620 20:18:28.164821 7651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:28.165576 7651 log.cc:824] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:28.167330 7651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f: No bootstrap required, opened a new log
I20260620 20:18:28.167629 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.163791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.2.235.254:42141)
0620 20:18:28.164335 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.164348 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.164392 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:18:28.164998 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.165014 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.165400 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.165698 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.166484 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.166488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.166499 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.166509 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.166555 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.166570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.166679 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.166772 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.166992 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260620 20:18:28.170892 7651 raft_consensus.cc:348] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42141 } }
I20260620 20:18:28.171092 7651 raft_consensus.cc:374] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:28.171165 7651 raft_consensus.cc:729] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f422a2858bdb4a1bbb5b32acdab39f9f, State: Initialized, Role: FOLLOWER
I20260620 20:18:28.171531 7651 consensus_queue.cc:260] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42141 } }
I20260620 20:18:28.171665 7651 raft_consensus.cc:388] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:28.171743 7651 raft_consensus.cc:482] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:28.171854 7651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:28.172590 7651 raft_consensus.cc:504] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42141 } }
I20260620 20:18:28.172896 7651 leader_election.cc:302] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f422a2858bdb4a1bbb5b32acdab39f9f; no voters:
I20260620 20:18:28.173188 7651 leader_election.cc:288] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:28.173301 7656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:28.173496 7656 raft_consensus.cc:686] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [term 1 LEADER]: Becoming Leader. State: Replica: f422a2858bdb4a1bbb5b32acdab39f9f, State: Running, Role: LEADER
I20260620 20:18:28.173782 7656 consensus_queue.cc:237] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42141 } }
I20260620 20:18:28.174240 7651 sys_catalog.cc:564] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:28.175246 7657 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42141 } } }
I20260620 20:18:28.175243 7658 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f422a2858bdb4a1bbb5b32acdab39f9f. Latest consensus state: current_term: 1 leader_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422a2858bdb4a1bbb5b32acdab39f9f" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42141 } } }
I20260620 20:18:28.175496 7658 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:28.175526 7657 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:28.176532 7665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:28.178849 7665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:28.183429 7665 catalog_manager.cc:1347] Generated new cluster ID: b5abfed694574420a86314bdab2dd7fa
I20260620 20:18:28.183481 7665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:28.206115 7665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:28.206405 7665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:28.212416 7665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f: Generated new TSK 0
I20260620 20:18:28.212941 7665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:28.223485 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42141
--builtin_ntp_servers=127.2.235.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:18:28.326539 7675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:28.326725 7675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:28.326757 7675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:28.330529 7675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:28.330578 7675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:28.330631 7675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:28.334771 7675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:28.335615 7675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:28.336781 7675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:28.344805 7681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:28.344842 7675 server_base.cc:1034] running on GCE node
W20260620 20:18:28.344854 7680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:28.344987 7683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:28.345414 7675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:28.346002 7675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:28.347179 7675 hybrid_clock.cc:648] HybridClock initialized: now 1781986708347172 us; error 29 us; skew 500 ppm
I20260620 20:18:28.348861 7675 webserver.cc:460] Webserver started at http://127.2.235.193:38917/ using document root <none> and password file <none>
I20260620 20:18:28.349416 7675 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:28.349514 7675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:28.349723 7675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:28.351508 7675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "55fb35ee6d014f808714b3592671ba2a"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "19ddf75998b1440aa3f754fd91900935"
server_key_iv: "bf6c863c86ca8c48d759ced764dd44e6"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.352020 7675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "55fb35ee6d014f808714b3592671ba2a"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "19ddf75998b1440aa3f754fd91900935"
server_key_iv: "bf6c863c86ca8c48d759ced764dd44e6"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.355234 7675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:18:28.357369 7689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.358436 7675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:28.358560 7675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "55fb35ee6d014f808714b3592671ba2a"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "19ddf75998b1440aa3f754fd91900935"
server_key_iv: "bf6c863c86ca8c48d759ced764dd44e6"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.358659 7675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:28.362484 7675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:28.362967 7675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:28.363126 7675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:28.363571 7675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:28.364308 7675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:28.364372 7675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.364429 7675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:28.364475 7675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.372432 7675 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:33095
I20260620 20:18:28.372455 7802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:33095 every 8 connection(s)
I20260620 20:18:28.373222 7675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:28.378273 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.374766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:38899 (local address 127.2.235.254:42141)
0620 20:18:28.374879 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.374882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.375946 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 20:18:28.377053 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.377056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.377087 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.377148 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.377963 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.377966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.377968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.377978 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.377995 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.378002 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.378069 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.378147 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.378178 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 20:18:28.379240 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7675
I20260620 20:18:28.379351 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:28.379585 2991 external_mini_cluster.cc:1428] Setting key 33f7dd73b29b6e2089dd7ed7bbba231f
I20260620 20:18:28.380267 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.375080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42141 (local address 127.2.235.193:38899)
0620 20:18:28.375812 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:28.375850 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:28.376904 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:28.377242 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:28.377261 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:28.377329 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:28.377731 (+ 402us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:28.377733 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:28.377761 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:28.377831 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:28.378053 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:28.378057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:28.378060 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:18:28.378160 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:18:28.378337 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":109,"threads_started":1}
I20260620 20:18:28.381232 7803 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42141
I20260620 20:18:28.381456 7803 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:28.381449 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42141
--builtin_ntp_servers=127.2.235.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:18:28.381827 7803 heartbeater.cc:511] Master 127.2.235.254:42141 requested a full tablet report, sending...
I20260620 20:18:28.383141 7615 ts_manager.cc:194] Registered new tserver with Master: 55fb35ee6d014f808714b3592671ba2a (127.2.235.193:33095)
W20260620 20:18:28.478892 7806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:28.479075 7806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:28.479106 7806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:28.482417 7806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:28.482453 7806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:28.482533 7806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:28.486656 7806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:28.487460 7806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:28.488536 7806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:28.496614 7806 server_base.cc:1034] running on GCE node
W20260620 20:18:28.496541 7814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:28.496554 7812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:28.496574 7811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:28.497166 7806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:28.497684 7806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:28.498828 7806 hybrid_clock.cc:648] HybridClock initialized: now 1781986708498806 us; error 37 us; skew 500 ppm
I20260620 20:18:28.500422 7806 webserver.cc:460] Webserver started at http://127.2.235.194:43001/ using document root <none> and password file <none>
I20260620 20:18:28.500895 7806 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:28.500950 7806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:28.501165 7806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:28.502902 7806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "ccd0ef278be0472faa98611e50410395"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "9ba7e4d6eda21bcefced81cc04df99a0"
server_key_iv: "5aa7280a4bdcee75d81acf10378d4a34"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.503437 7806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "ccd0ef278be0472faa98611e50410395"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "9ba7e4d6eda21bcefced81cc04df99a0"
server_key_iv: "5aa7280a4bdcee75d81acf10378d4a34"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.506820 7806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:28.508874 7820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.509999 7806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:28.510110 7806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "ccd0ef278be0472faa98611e50410395"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "9ba7e4d6eda21bcefced81cc04df99a0"
server_key_iv: "5aa7280a4bdcee75d81acf10378d4a34"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.510203 7806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:28.514205 7806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:28.514695 7806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:28.514858 7806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:28.515352 7806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:28.516144 7806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:28.516214 7806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.516278 7806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:28.516327 7806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.524185 7806 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:43547
I20260620 20:18:28.524202 7933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:43547 every 8 connection(s)
I20260620 20:18:28.524884 7806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:28.526678 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7806
I20260620 20:18:28.526789 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:28.527055 2991 external_mini_cluster.cc:1428] Setting key b18dcefcc78831e4d6c7abe62ef5b38a
I20260620 20:18:28.529357 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42141
--builtin_ntp_servers=127.2.235.212:41301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:18:28.531322 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.526547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:38701 (local address 127.2.235.254:42141)
0620 20:18:28.526667 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.526670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.528008 (+ 1338us) server_negotiation.cc:407] Connection header received
0620 20:18:28.529364 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.529367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.529404 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.529513 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.530744 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.530751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.530755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.530775 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.530807 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.530823 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.530946 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.531093 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.531143 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:28.531976 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.526940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42141 (local address 127.2.235.194:38701)
0620 20:18:28.527885 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:28.527921 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:28.529135 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:28.529726 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:28.529737 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:28.529839 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:28.530407 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:28.530412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:28.530464 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:28.530549 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:28.530938 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:28.530945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:28.530949 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:18:28.531093 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 20:18:28.531260 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":135,"threads_started":1}
I20260620 20:18:28.533106 7934 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42141
I20260620 20:18:28.533329 7934 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:28.533697 7934 heartbeater.cc:511] Master 127.2.235.254:42141 requested a full tablet report, sending...
I20260620 20:18:28.534652 7615 ts_manager.cc:194] Registered new tserver with Master: ccd0ef278be0472faa98611e50410395 (127.2.235.194:43547)
W20260620 20:18:28.629809 7937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:28.630015 7937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:28.630064 7937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:28.633850 7937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:28.633893 7937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:28.633972 7937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:28.638463 7937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:28.639349 7937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:28.640542 7937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:28.649180 7937 server_base.cc:1034] running on GCE node
W20260620 20:18:28.649114 7945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:28.649135 7942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:28.649129 7943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:28.649703 7937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:28.650239 7937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:28.651397 7937 hybrid_clock.cc:648] HybridClock initialized: now 1781986708651378 us; error 26 us; skew 500 ppm
I20260620 20:18:28.653033 7937 webserver.cc:460] Webserver started at http://127.2.235.195:43309/ using document root <none> and password file <none>
I20260620 20:18:28.653592 7937 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:28.653651 7937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:28.653842 7937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:28.655673 7937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "400d8f77102a43d1b0b0bebb39133c4c"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "4c91b09b22f4330223bfccf664cccfd4"
server_key_iv: "2d5763458f9b65bb1c84b3db89e89254"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.656201 7937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "400d8f77102a43d1b0b0bebb39133c4c"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "4c91b09b22f4330223bfccf664cccfd4"
server_key_iv: "2d5763458f9b65bb1c84b3db89e89254"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.659584 7937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 20:18:28.661759 7951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.662906 7937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 20:18:28.663048 7937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "400d8f77102a43d1b0b0bebb39133c4c"
format_stamp: "Formatted at 2026-06-20 20:18:28 on dist-test-slave-8z53"
server_key: "4c91b09b22f4330223bfccf664cccfd4"
server_key_iv: "2d5763458f9b65bb1c84b3db89e89254"
server_key_version: "encryptionkey@0"
I20260620 20:18:28.663158 7937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:28.667387 7937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:28.667906 7937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:28.668081 7937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:28.668551 7937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:28.669427 7937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:28.669493 7937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.669564 7937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:28.669605 7937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:28.678665 7937 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:40475
I20260620 20:18:28.678695 8064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:40475 every 8 connection(s)
I20260620 20:18:28.679423 7937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:28.684540 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.680927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:55637 (local address 127.2.235.254:42141)
0620 20:18:28.681048 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.681052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.681942 (+ 890us) server_negotiation.cc:407] Connection header received
0620 20:18:28.683057 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.683061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.683098 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.683197 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.684086 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.684089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.684091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.684102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.684120 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.684135 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.684218 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.684353 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.684392 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:28.685325 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.681235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42141 (local address 127.2.235.195:55637)
0620 20:18:28.681823 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:28.681854 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:28.682870 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:28.683336 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:28.683342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:28.683415 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:28.683859 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:28.683861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:28.683893 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:28.683949 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:28.684213 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:28.684222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:28.684229 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:18:28.684377 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 20:18:28.684592 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260620 20:18:28.685544 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 7937
I20260620 20:18:28.685662 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:28.685928 2991 external_mini_cluster.cc:1428] Setting key 66bb9ab108de19280995e6dc4ee6e5fe
I20260620 20:18:28.686436 8065 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42141
I20260620 20:18:28.686646 8065 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:28.687031 8065 heartbeater.cc:511] Master 127.2.235.254:42141 requested a full tablet report, sending...
I20260620 20:18:28.688040 7615 ts_manager.cc:194] Registered new tserver with Master: 400d8f77102a43d1b0b0bebb39133c4c (127.2.235.195:40475)
I20260620 20:18:28.699225 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:28.702243 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.700790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.2.235.254:42141)
0620 20:18:28.700909 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.700913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.701090 (+ 177us) server_negotiation.cc:407] Connection header received
0620 20:18:28.701218 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.701221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.701277 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.701339 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.701892 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.701895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.701897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.701907 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.701923 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.701931 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.702019 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.702103 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.702136 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:28.704489 7615 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:28.707285 7615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:28.717696 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.714479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:33095 (local address 127.0.0.1:60984)
0620 20:18:28.714774 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:28.714820 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:28.714910 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:28.716221 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:28.716226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:28.716315 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:28.717004 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:28.717006 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:28.717014 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:28.717033 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:28.717360 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:28.717368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:28.717374 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:18:28.717490 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 20:18:28.717563 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260620 20:18:28.717715 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.714885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:43547 (local address 127.0.0.1:40024)
0620 20:18:28.715801 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:28.715815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:28.715901 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:28.716307 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:28.716309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:28.716320 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:28.716899 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:28.716902 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:28.716920 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:28.717004 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:28.717422 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:28.717425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:28.717428 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:18:28.717505 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:18:28.717571 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260620 20:18:28.717725 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.714591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.2.235.193:33095)
0620 20:18:28.715149 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.715162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.715205 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:18:28.715259 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.715264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.715447 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.715611 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.717155 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.717160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.717172 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.717184 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.717225 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.717283 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.717380 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.717503 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.717574 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":104,"threads_started":1}
I20260620 20:18:28.718147 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.715252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.2.235.194:43547)
0620 20:18:28.715679 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.715690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.715911 (+ 221us) server_negotiation.cc:407] Connection header received
0620 20:18:28.715991 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.716006 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.716176 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.716302 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.717156 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.717161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.717172 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.717184 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.717225 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.717284 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.717380 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.717548 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.717612 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":85,"threads_started":1}
I20260620 20:18:28.718734 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.715009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40475 (local address 127.0.0.1:50502)
0620 20:18:28.715902 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:28.715916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:28.715990 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:28.716909 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:28.716912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:28.716933 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:28.718111 (+ 1178us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:28.718112 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:28.718121 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:28.718145 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:28.718532 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:28.718536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:28.718538 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 20:18:28.718604 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:18:28.718633 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":74,"threads_started":1}
I20260620 20:18:28.718832 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:28.715357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50502 (local address 127.2.235.195:40475)
0620 20:18:28.715749 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:28.715760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:28.716072 (+ 312us) server_negotiation.cc:407] Connection header received
0620 20:18:28.716116 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:28.716120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:28.716248 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:28.716351 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:28.718310 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:28.718315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:28.718325 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:28.718341 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:28.718376 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:28.718399 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:28.718484 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:28.718644 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:18:28.718699 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":138,"threads_started":1}
I20260620 20:18:28.720319 7868 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf462e74088478fa569b134ec5eb27a (DEFAULT_TABLE table=test-table [id=2a85122d1a804e01ba40683f257a8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:28.720319 7737 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf462e74088478fa569b134ec5eb27a (DEFAULT_TABLE table=test-table [id=2a85122d1a804e01ba40683f257a8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:28.721051 7868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf462e74088478fa569b134ec5eb27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:28.721072 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf462e74088478fa569b134ec5eb27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:28.726706 8079 tablet_bootstrap.cc:492] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: Bootstrap starting.
I20260620 20:18:28.727579 7999 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf462e74088478fa569b134ec5eb27a (DEFAULT_TABLE table=test-table [id=2a85122d1a804e01ba40683f257a8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:28.728261 7999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf462e74088478fa569b134ec5eb27a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:28.728629 8079 tablet_bootstrap.cc:654] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:28.729230 8079 log.cc:824] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:28.729576 8080 tablet_bootstrap.cc:492] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Bootstrap starting.
I20260620 20:18:28.731289 8079 tablet_bootstrap.cc:492] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: No bootstrap required, opened a new log
I20260620 20:18:28.731474 8079 ts_tablet_manager.cc:1397] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:28.731600 8080 tablet_bootstrap.cc:654] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:28.732353 8080 log.cc:824] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:28.733731 8083 tablet_bootstrap.cc:492] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Bootstrap starting.
I20260620 20:18:28.734248 8080 tablet_bootstrap.cc:492] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: No bootstrap required, opened a new log
I20260620 20:18:28.734423 8080 ts_tablet_manager.cc:1397] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:18:28.735673 8083 tablet_bootstrap.cc:654] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:28.735845 8079 raft_consensus.cc:348] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:28.736092 8079 raft_consensus.cc:374] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:28.736156 8079 raft_consensus.cc:729] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fb35ee6d014f808714b3592671ba2a, State: Initialized, Role: FOLLOWER
I20260620 20:18:28.736496 8083 log.cc:824] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:28.736594 8079 consensus_queue.cc:260] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:28.737227 7803 heartbeater.cc:503] Master 127.2.235.254:42141 was elected leader, sending a full tablet report...
I20260620 20:18:28.738016 8083 tablet_bootstrap.cc:492] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: No bootstrap required, opened a new log
I20260620 20:18:28.738195 8083 ts_tablet_manager.cc:1397] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:28.738478 8079 ts_tablet_manager.cc:1428] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 20:18:28.738430 8080 raft_consensus.cc:348] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:28.738631 8080 raft_consensus.cc:374] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:28.738688 8080 raft_consensus.cc:729] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd0ef278be0472faa98611e50410395, State: Initialized, Role: FOLLOWER
I20260620 20:18:28.739108 8080 consensus_queue.cc:260] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:28.739639 7934 heartbeater.cc:503] Master 127.2.235.254:42141 was elected leader, sending a full tablet report...
I20260620 20:18:28.739902 8080 ts_tablet_manager.cc:1428] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:18:28.741686 8083 raft_consensus.cc:348] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:28.741909 8083 raft_consensus.cc:374] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:28.742296 8083 raft_consensus.cc:729] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400d8f77102a43d1b0b0bebb39133c4c, State: Initialized, Role: FOLLOWER
I20260620 20:18:28.742653 8083 consensus_queue.cc:260] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:28.743162 8065 heartbeater.cc:503] Master 127.2.235.254:42141 was elected leader, sending a full tablet report...
I20260620 20:18:28.743379 8083 ts_tablet_manager.cc:1428] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260620 20:18:28.775799 7935 tablet.cc:2369] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:28.874217 7804 tablet.cc:2369] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:18:28.930289 8066 tablet.cc:2369] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:29.122727 8085 raft_consensus.cc:482] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:29.122910 8085 raft_consensus.cc:504] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:29.123968 8085 leader_election.cc:288] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 400d8f77102a43d1b0b0bebb39133c4c (127.2.235.195:40475), ccd0ef278be0472faa98611e50410395 (127.2.235.194:43547)
I20260620 20:18:29.126147 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.124203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:43547 (local address 127.2.235.193:37033)
0620 20:18:29.124558 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:29.124570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:29.124709 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:29.124984 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:29.124987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:29.125002 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:29.125604 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:29.125607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:29.125620 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:29.125657 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:29.125921 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:29.125923 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:29.125926 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:18:29.125993 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:18:29.126024 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"spinlock_wait_cycles":39680,"thread_start_us":122,"threads_started":1}
I20260620 20:18:29.126168 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.124289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:37033 (local address 127.2.235.194:43547)
0620 20:18:29.124437 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.124442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.124644 (+ 202us) server_negotiation.cc:407] Connection header received
0620 20:18:29.124833 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.124836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.124872 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.124959 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:29.125779 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:29.125785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:29.125788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:29.125801 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:29.125822 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:29.125832 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:29.125931 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:29.125995 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:18:29.126035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:18:29.126464 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.124287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:59761 (local address 127.2.235.195:40475)
0620 20:18:29.124437 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.124442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.124661 (+ 219us) server_negotiation.cc:407] Connection header received
0620 20:18:29.124812 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.124817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.124886 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.124961 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:29.126003 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:29.126006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:29.126008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:29.126020 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:29.126040 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:29.126050 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:29.126110 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:29.126294 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:18:29.126345 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:18:29.126749 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.124201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40475 (local address 127.2.235.193:59761)
0620 20:18:29.124561 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:29.124572 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:29.124710 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:29.125054 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:29.125059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:29.125076 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:29.125806 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:29.125808 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:29.125818 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:29.125841 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:29.126178 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:29.126184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:29.126190 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:18:29.126612 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 20:18:29.126649 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":67,"spinlock_wait_cycles":14720,"thread_start_us":90,"threads_started":1}
I20260620 20:18:29.126772 7888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf462e74088478fa569b134ec5eb27a" candidate_uuid: "55fb35ee6d014f808714b3592671ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd0ef278be0472faa98611e50410395" is_pre_election: true
I20260620 20:18:29.127036 7888 raft_consensus.cc:2454] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fb35ee6d014f808714b3592671ba2a in term 0.
I20260620 20:18:29.127351 8019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf462e74088478fa569b134ec5eb27a" candidate_uuid: "55fb35ee6d014f808714b3592671ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400d8f77102a43d1b0b0bebb39133c4c" is_pre_election: true
I20260620 20:18:29.127499 7691 leader_election.cc:302] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fb35ee6d014f808714b3592671ba2a, ccd0ef278be0472faa98611e50410395; no voters:
I20260620 20:18:29.127617 8019 raft_consensus.cc:2454] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fb35ee6d014f808714b3592671ba2a in term 0.
I20260620 20:18:29.127779 8085 raft_consensus.cc:2781] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:29.127889 8085 raft_consensus.cc:482] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:29.127944 8085 raft_consensus.cc:3037] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:29.128991 8085 raft_consensus.cc:504] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:29.129302 8085 leader_election.cc:288] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [CANDIDATE]: Term 1 election: Requested vote from peers 400d8f77102a43d1b0b0bebb39133c4c (127.2.235.195:40475), ccd0ef278be0472faa98611e50410395 (127.2.235.194:43547)
I20260620 20:18:29.129560 8019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf462e74088478fa569b134ec5eb27a" candidate_uuid: "55fb35ee6d014f808714b3592671ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400d8f77102a43d1b0b0bebb39133c4c"
I20260620 20:18:29.129676 8019 raft_consensus.cc:3037] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:29.129680 7888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf462e74088478fa569b134ec5eb27a" candidate_uuid: "55fb35ee6d014f808714b3592671ba2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd0ef278be0472faa98611e50410395"
I20260620 20:18:29.129820 7888 raft_consensus.cc:3037] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:29.130743 8019 raft_consensus.cc:2454] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fb35ee6d014f808714b3592671ba2a in term 1.
I20260620 20:18:29.130884 7888 raft_consensus.cc:2454] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fb35ee6d014f808714b3592671ba2a in term 1.
I20260620 20:18:29.131067 7690 leader_election.cc:302] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 400d8f77102a43d1b0b0bebb39133c4c, 55fb35ee6d014f808714b3592671ba2a; no voters:
I20260620 20:18:29.131278 8085 raft_consensus.cc:2781] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:29.131494 8085 raft_consensus.cc:686] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 1 LEADER]: Becoming Leader. State: Replica: 55fb35ee6d014f808714b3592671ba2a, State: Running, Role: LEADER
I20260620 20:18:29.131794 8085 consensus_queue.cc:237] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } }
I20260620 20:18:29.134686 7615 catalog_manager.cc:5495] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fb35ee6d014f808714b3592671ba2a (127.2.235.193). New cstate: current_term: 1 leader_uuid: "55fb35ee6d014f808714b3592671ba2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fb35ee6d014f808714b3592671ba2a" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 33095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:29.162632 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.161189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.2.235.193:33095)
0620 20:18:29.161309 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.161314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.161327 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:29.161474 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.161476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.161523 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.161583 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:29.162154 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:29.162157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:29.162158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:29.162167 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:29.162182 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:29.162189 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:29.162248 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:29.162365 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 20:18:29.162479 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 20:18:29.168337 7888 raft_consensus.cc:1261] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 1 FOLLOWER]: Refusing update from remote peer 55fb35ee6d014f808714b3592671ba2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:29.168349 8019 raft_consensus.cc:1261] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 1 FOLLOWER]: Refusing update from remote peer 55fb35ee6d014f808714b3592671ba2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:29.168944 8085 consensus_queue.cc:1035] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "400d8f77102a43d1b0b0bebb39133c4c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:29.169173 8090 consensus_queue.cc:1035] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccd0ef278be0472faa98611e50410395" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 43547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:29.175757 8095 mvcc.cc:205] Tried to move back new op lower bound from 7299017560747929600 to 7299017560606142464. Current Snapshot: MvccSnapshot[applied={T|T < 7299017560747929600}]
I20260620 20:18:29.176992 8094 mvcc.cc:205] Tried to move back new op lower bound from 7299017560747929600 to 7299017560606142464. Current Snapshot: MvccSnapshot[applied={T|T < 7299017560747929600}]
I20260620 20:18:29.183317 7614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:29.183447 7614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:29.185642 7614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f422a2858bdb4a1bbb5b32acdab39f9f: Sending DeleteTablet for 3 replicas of tablet 6bf462e74088478fa569b134ec5eb27a
I20260620 20:18:29.186329 7999 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf462e74088478fa569b134ec5eb27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:29 UTC) from {username='slave'} at 127.0.0.1:50502
I20260620 20:18:29.186369 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf462e74088478fa569b134ec5eb27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:29 UTC) from {username='slave'} at 127.0.0.1:60984
I20260620 20:18:29.186738 8103 tablet_replica.cc:330] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: stopping tablet replica
I20260620 20:18:29.186755 7868 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf462e74088478fa569b134ec5eb27a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:29 UTC) from {username='slave'} at 127.0.0.1:40024
I20260620 20:18:29.187049 8104 tablet_replica.cc:330] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a: stopping tablet replica
I20260620 20:18:29.187158 8103 raft_consensus.cc:2229] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:29.187290 8104 raft_consensus.cc:2229] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:29.187474 8103 raft_consensus.cc:2258] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:29.187541 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7675
I20260620 20:18:29.187582 8104 raft_consensus.cc:2258] T 6bf462e74088478fa569b134ec5eb27a P 55fb35ee6d014f808714b3592671ba2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:29.189167 8105 tablet_replica.cc:330] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: stopping tablet replica
I20260620 20:18:29.189430 8105 raft_consensus.cc:2229] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:29.189705 8105 raft_consensus.cc:2258] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:29.190037 8103 ts_tablet_manager.cc:1905] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:29.192343 8105 ts_tablet_manager.cc:1905] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:29.193360 8103 ts_tablet_manager.cc:1918] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:29.193475 8103 log.cc:1196] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/6bf462e74088478fa569b134ec5eb27a
I20260620 20:18:29.193790 8103 ts_tablet_manager.cc:1939] T 6bf462e74088478fa569b134ec5eb27a P 400d8f77102a43d1b0b0bebb39133c4c: Deleting consensus metadata
I20260620 20:18:29.194591 8105 ts_tablet_manager.cc:1918] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:29.194695 8105 log.cc:1196] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/6bf462e74088478fa569b134ec5eb27a
I20260620 20:18:29.194892 7599 catalog_manager.cc:4841] TS 400d8f77102a43d1b0b0bebb39133c4c (127.2.235.195:40475): tablet 6bf462e74088478fa569b134ec5eb27a (table test-table [id=2a85122d1a804e01ba40683f257a8a36]) successfully deleted
I20260620 20:18:29.195035 8105 ts_tablet_manager.cc:1939] T 6bf462e74088478fa569b134ec5eb27a P ccd0ef278be0472faa98611e50410395: Deleting consensus metadata
I20260620 20:18:29.195928 7600 catalog_manager.cc:4841] TS ccd0ef278be0472faa98611e50410395 (127.2.235.194:43547): tablet 6bf462e74088478fa569b134ec5eb27a (table test-table [id=2a85122d1a804e01ba40683f257a8a36]) successfully deleted
W20260620 20:18:29.197491 7602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:33095 (error 108)
I20260620 20:18:29.198215 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7806
I20260620 20:18:29.198365 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.198094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:33095 (local address 127.0.0.1:32776)
0620 20:18:29.198211 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:29.198282 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:33095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 20:18:29.198521 7602 catalog_manager.cc:4584] TS 55fb35ee6d014f808714b3592671ba2a (127.2.235.193:33095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bf462e74088478fa569b134ec5eb27a: Network error: Client connection negotiation failed: client connection to 127.2.235.193:33095: connect: Connection refused (error 111)
I20260620 20:18:29.204666 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7937
I20260620 20:18:29.209740 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 7584
2026-06-20T20:18:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1219 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 20:18:29.230280 2991 mini_oidc.cc:152] Starting JWKS server
I20260620 20:18:29.231698 2991 webserver.cc:460] Webserver started at https://127.0.0.1:39813/ using document root <none> and password file <none>
I20260620 20:18:29.231865 2991 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:18:29.232117 2991 webserver.cc:460] Webserver started at http://127.0.0.1:32855/ using document root <none> and password file <none>
2026-06-20T20:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:29Z Disabled control of system clock
I20260620 20:18:29.239758 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:38179
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:38179
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39813/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/testchainca.cert with env {}
W20260620 20:18:29.340200 8112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:29.340471 8112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:29.340545 8112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:29.344266 8112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:29.344312 8112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:29.344326 8112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:29.344341 8112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:29.344355 8112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:18:29.344369 8112 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:18:29.344430 8112 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39813/jwks.json
I20260620 20:18:29.349363 8112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:38179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39813/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:29.350277 8112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:29.351209 8112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:29.358948 8120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:29.359019 8112 server_base.cc:1034] running on GCE node
W20260620 20:18:29.358964 8118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:29.358970 8117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:29.359622 8112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:29.360646 8112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:29.361874 8112 hybrid_clock.cc:648] HybridClock initialized: now 1781986709361857 us; error 55 us; skew 500 ppm
I20260620 20:18:29.363584 8112 webserver.cc:460] Webserver started at http://127.2.235.254:40233/ using document root <none> and password file <none>
I20260620 20:18:29.364123 8112 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:29.364204 8112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:29.364418 8112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:29.366348 8112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "f670920d25664c88986b3cdbb8838e36"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "c798c4fee672409fbd982da196aaf31e"
server_key_iv: "0f722e5a51a57e50e85fd94fcfc7ebba"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.366917 8112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "f670920d25664c88986b3cdbb8838e36"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "c798c4fee672409fbd982da196aaf31e"
server_key_iv: "0f722e5a51a57e50e85fd94fcfc7ebba"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.370142 8112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:29.372164 8126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:29.373236 8112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:29.373417 8112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "f670920d25664c88986b3cdbb8838e36"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "c798c4fee672409fbd982da196aaf31e"
server_key_iv: "0f722e5a51a57e50e85fd94fcfc7ebba"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.373512 8112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:29.383693 8112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:29.384215 8112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:29.384404 8112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:29.390658 8112 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:38179
I20260620 20:18:29.390698 8178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:38179 every 8 connection(s)
I20260620 20:18:29.391407 8112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:29.393910 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:29.394001 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8112
I20260620 20:18:29.394116 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:29.394366 2991 external_mini_cluster.cc:1428] Setting key edb2eed4cc586ab597b2078bbc80d934
I20260620 20:18:29.400156 8179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36: Bootstrap starting.
I20260620 20:18:29.402554 8179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:29.403254 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.395468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.2.235.254:38179)
0620 20:18:29.396095 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.396113 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.396154 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:18:29.396933 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.396954 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.397400 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.397717 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:29.398081 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.399367 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.400110 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.400701 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.400844 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.401625 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:29.401631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:29.401654 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:29.402029 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:29.402088 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:29.402111 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:29.402189 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:29.402278 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 20:18:29.402503 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260620 20:18:29.403275 8179 log.cc:824] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:29.404968 8179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36: No bootstrap required, opened a new log
I20260620 20:18:29.405715 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:18:29.408313 8179 raft_consensus.cc:348] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670920d25664c88986b3cdbb8838e36" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38179 } }
I20260620 20:18:29.408509 8179 raft_consensus.cc:374] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:29.408615 8179 raft_consensus.cc:729] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f670920d25664c88986b3cdbb8838e36, State: Initialized, Role: FOLLOWER
I20260620 20:18:29.409065 8179 consensus_queue.cc:260] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670920d25664c88986b3cdbb8838e36" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38179 } }
I20260620 20:18:29.409210 8179 raft_consensus.cc:388] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:29.409313 8179 raft_consensus.cc:482] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:29.409406 8179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:29.410369 8179 raft_consensus.cc:504] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670920d25664c88986b3cdbb8838e36" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38179 } }
I20260620 20:18:29.410682 8179 leader_election.cc:302] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f670920d25664c88986b3cdbb8838e36; no voters:
I20260620 20:18:29.410957 8179 leader_election.cc:288] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:29.411068 8184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:29.411260 8184 raft_consensus.cc:686] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [term 1 LEADER]: Becoming Leader. State: Replica: f670920d25664c88986b3cdbb8838e36, State: Running, Role: LEADER
I20260620 20:18:29.411547 8184 consensus_queue.cc:237] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670920d25664c88986b3cdbb8838e36" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38179 } }
I20260620 20:18:29.411998 8179 sys_catalog.cc:564] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:29.412962 8185 sys_catalog.cc:455] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f670920d25664c88986b3cdbb8838e36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670920d25664c88986b3cdbb8838e36" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38179 } } }
I20260620 20:18:29.413080 8185 sys_catalog.cc:458] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:29.413277 8186 sys_catalog.cc:455] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f670920d25664c88986b3cdbb8838e36. Latest consensus state: current_term: 1 leader_uuid: "f670920d25664c88986b3cdbb8838e36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f670920d25664c88986b3cdbb8838e36" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38179 } } }
I20260620 20:18:29.413376 8186 sys_catalog.cc:458] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36 [sys.catalog]: This master's current role is: LEADER
W20260620 20:18:29.416383 8199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:29.416465 8199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:29.416532 8193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:29.417097 8193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:29.421636 8193 catalog_manager.cc:1347] Generated new cluster ID: 65bba915df04491b8744b923508dd0d3
I20260620 20:18:29.421690 8193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:29.440403 8193 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:29.441147 8193 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:29.450569 8193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f670920d25664c88986b3cdbb8838e36: Generated new TSK 0
I20260620 20:18:29.451114 8193 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:18:29.480749 8209 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 20:18:29.480831 8209 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:18:29.481312 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.464827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.2.235.254:38179)
0620 20:18:29.464943 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.464949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.465116 (+ 167us) server_negotiation.cc:407] Connection header received
0620 20:18:29.465318 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.465321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.465372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.465484 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:18:29.465697 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.466538 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.467371 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.467707 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.467796 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.468064 (+ 268us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:18:29.480969 (+ 12905us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:18:29.481179 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:18:29.481375 8182 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 20:18:29.482240 8208 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:29.464707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:38179 (local address 127.0.0.1:38242)
0620 20:18:29.465044 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:29.465056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:29.465151 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:29.465496 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:29.465498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:29.465504 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:18:29.465599 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:29.465603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.466661 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.466664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:29.467254 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:29.467262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.467801 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.467803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:29.467878 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.467913 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:18:29.481115 (+ 13202us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:18:29.481317 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.235.254:38179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.235.254:38179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 20:18:29.482952 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8112
2026-06-20T20:18:29Z chronyd exiting
I20260620 20:18:29.519611 2991 test_util.cc:175] -----------------------------------------------
I20260620 20:18:29.519665 2991 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986686246774-2991-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (290 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 20:18:29.520502 2991 mini_oidc.cc:152] Starting JWKS server
I20260620 20:18:29.521198 2991 webserver.cc:460] Webserver started at https://127.0.0.1:33107/ using document root <none> and password file <none>
I20260620 20:18:29.521325 2991 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:18:29.521556 2991 webserver.cc:460] Webserver started at http://127.0.0.1:43717/ using document root <none> and password file <none>
2026-06-20T20:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:29Z Disabled control of system clock
I20260620 20:18:29.528884 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:37515
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:37515
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33107/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/testchainca.cert with env {}
W20260620 20:18:29.626370 8215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:29.626633 8215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:29.626696 8215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:29.630332 8215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:29.630401 8215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:29.630427 8215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:29.630465 8215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:29.630491 8215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:18:29.630522 8215 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:18:29.630551 8215 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33107/jwks.json
I20260620 20:18:29.635190 8215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:37515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:37515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33107/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:29.636086 8215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:29.636960 8215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:29.644201 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:29.644304 8215 server_base.cc:1034] running on GCE node
W20260620 20:18:29.644311 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:29.644484 8223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:29.644955 8215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:29.645954 8215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:29.647126 8215 hybrid_clock.cc:648] HybridClock initialized: now 1781986709647107 us; error 35 us; skew 500 ppm
I20260620 20:18:29.648711 8215 webserver.cc:460] Webserver started at http://127.2.235.254:44283/ using document root <none> and password file <none>
I20260620 20:18:29.649195 8215 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:29.649294 8215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:29.649518 8215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:29.651278 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "93e578d61d6547c9b12cd479cb52259b"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "12719668d3c4c0c5dc8386ff2ee49ed0"
server_key_iv: "e821ef2931f53a82bc567ae1e268880f"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.651779 8215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "93e578d61d6547c9b12cd479cb52259b"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "12719668d3c4c0c5dc8386ff2ee49ed0"
server_key_iv: "e821ef2931f53a82bc567ae1e268880f"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.655017 8215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:29.657194 8229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:29.658422 8215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:29.658579 8215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "93e578d61d6547c9b12cd479cb52259b"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "12719668d3c4c0c5dc8386ff2ee49ed0"
server_key_iv: "e821ef2931f53a82bc567ae1e268880f"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.658694 8215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:29.679518 8215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:29.680094 8215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:29.680284 8215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:29.687076 8215 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:37515
I20260620 20:18:29.687084 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:37515 every 8 connection(s)
I20260620 20:18:29.687882 8215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:29.690881 8282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:29.693470 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8215
I20260620 20:18:29.693583 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:29.693821 2991 external_mini_cluster.cc:1428] Setting key 385bbc42f9eeeaeff6a9acd504ceb4fa
I20260620 20:18:29.696141 8282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b: Bootstrap starting.
I20260620 20:18:29.698180 8282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:29.698948 8282 log.cc:824] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:29.700752 8282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b: No bootstrap required, opened a new log
I20260620 20:18:29.702211 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.694964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.2.235.254:37515)
0620 20:18:29.695544 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.695557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.695589 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:18:29.696185 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.696200 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.696653 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.696977 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:29.697402 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.698238 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.699306 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.699812 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.699919 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.700657 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:29.700662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:29.700672 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:29.701052 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:29.701099 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:29.701117 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:29.701181 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:29.701347 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 20:18:29.701617 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260620 20:18:29.704205 8282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e578d61d6547c9b12cd479cb52259b" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37515 } }
I20260620 20:18:29.704406 8282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:29.704473 8282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e578d61d6547c9b12cd479cb52259b, State: Initialized, Role: FOLLOWER
I20260620 20:18:29.704826 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:18:29.704873 8282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e578d61d6547c9b12cd479cb52259b" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37515 } }
I20260620 20:18:29.705001 8282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:29.705049 8282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:29.705127 8282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:29.706156 8282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e578d61d6547c9b12cd479cb52259b" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37515 } }
I20260620 20:18:29.706493 8282 leader_election.cc:302] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93e578d61d6547c9b12cd479cb52259b; no voters:
I20260620 20:18:29.706810 8282 leader_election.cc:288] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:29.706923 8287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:29.707172 8287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [term 1 LEADER]: Becoming Leader. State: Replica: 93e578d61d6547c9b12cd479cb52259b, State: Running, Role: LEADER
I20260620 20:18:29.707434 8287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e578d61d6547c9b12cd479cb52259b" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37515 } }
I20260620 20:18:29.707828 8282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:29.708776 8289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e578d61d6547c9b12cd479cb52259b. Latest consensus state: current_term: 1 leader_uuid: "93e578d61d6547c9b12cd479cb52259b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e578d61d6547c9b12cd479cb52259b" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37515 } } }
I20260620 20:18:29.708902 8289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:29.709272 8296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:29.709209 8288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93e578d61d6547c9b12cd479cb52259b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e578d61d6547c9b12cd479cb52259b" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37515 } } }
I20260620 20:18:29.709365 8288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:29.712141 8296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:29.717291 8296 catalog_manager.cc:1347] Generated new cluster ID: d2c9d67283664e0b91c7f3375174636b
I20260620 20:18:29.717346 8296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:29.723848 8296 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:29.724603 8296 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:29.729414 8296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93e578d61d6547c9b12cd479cb52259b: Generated new TSK 0
I20260620 20:18:29.729954 8296 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:18:29.758648 8312 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 20:18:29.758708 8312 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:18:29.759109 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.738908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.2.235.254:37515)
0620 20:18:29.739001 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.739003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.739159 (+ 156us) server_negotiation.cc:407] Connection header received
0620 20:18:29.739274 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.739276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.739312 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.739401 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:18:29.739611 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.740025 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.741164 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.741415 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.741501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.741781 (+ 280us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:18:29.758873 (+ 17092us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:18:29.759028 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 20:18:29.759174 8285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49996: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 20:18:29.759280 8311 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:29.738825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37515 (local address 127.0.0.1:49996)
0620 20:18:29.739087 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:29.739098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:29.739188 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:29.739413 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:29.739416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:29.739423 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:18:29.739519 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:29.739525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.740125 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.740128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:29.741066 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:29.741075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.741529 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.741533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:29.741614 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.741639 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:18:29.759002 (+ 17363us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:18:29.759129 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.235.254:37515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260620 20:18:29.759824 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8215
2026-06-20T20:18:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (278 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 20:18:29.799019 2991 mini_oidc.cc:152] Starting JWKS server
I20260620 20:18:29.799782 2991 webserver.cc:460] Webserver started at https://127.0.0.1:41795/ using document root <none> and password file <none>
I20260620 20:18:29.799898 2991 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:18:29.800155 2991 webserver.cc:460] Webserver started at http://127.0.0.1:39129/ using document root <none> and password file <none>
2026-06-20T20:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:29Z Disabled control of system clock
I20260620 20:18:29.808104 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34515
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:43049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:34515
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41795/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:18:29.908274 8318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:29.908511 8318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:29.908573 8318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:29.912750 8318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:29.912829 8318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:29.912871 8318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:29.912906 8318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:29.912957 8318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:18:29.912997 8318 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:18:29.913030 8318 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41795/jwks.json
I20260620 20:18:29.917568 8318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:43049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:34515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41795/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:29.918416 8318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:29.919634 8318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:29.927371 8324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:29.927378 8323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:29.927500 8318 server_base.cc:1034] running on GCE node
W20260620 20:18:29.927661 8326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:29.928140 8318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:29.929014 8318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:29.930202 8318 hybrid_clock.cc:648] HybridClock initialized: now 1781986709930165 us; error 43 us; skew 500 ppm
I20260620 20:18:29.935307 8318 webserver.cc:460] Webserver started at http://127.2.235.254:36807/ using document root <none> and password file <none>
I20260620 20:18:29.935848 8318 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:29.935957 8318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:29.936172 8318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:29.938061 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "73c840a2a5444f9db4816afbe9ac0338"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "fe71ded938ce770f4167a0eec9ba1fe8"
server_key_iv: "3cfd97378e219793dcb83a9e1853c6f3"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.938546 8318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "73c840a2a5444f9db4816afbe9ac0338"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "fe71ded938ce770f4167a0eec9ba1fe8"
server_key_iv: "3cfd97378e219793dcb83a9e1853c6f3"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.941776 8318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:29.943845 8332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:29.944906 8318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:29.945032 8318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "73c840a2a5444f9db4816afbe9ac0338"
format_stamp: "Formatted at 2026-06-20 20:18:29 on dist-test-slave-8z53"
server_key: "fe71ded938ce770f4167a0eec9ba1fe8"
server_key_iv: "3cfd97378e219793dcb83a9e1853c6f3"
server_key_version: "encryptionkey@0"
I20260620 20:18:29.945132 8318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:29.953791 8318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:29.954289 8318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:29.954450 8318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:29.960474 8318 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:34515
I20260620 20:18:29.960497 8384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:34515 every 8 connection(s)
I20260620 20:18:29.961244 8318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:29.962782 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8318
I20260620 20:18:29.962904 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:29.963145 2991 external_mini_cluster.cc:1428] Setting key d45bf4f312e45d256b4d8ac4e39035c2
I20260620 20:18:29.963975 8385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:29.969674 8385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338: Bootstrap starting.
I20260620 20:18:29.971246 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:29.964292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.2.235.254:34515)
0620 20:18:29.964806 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:29.964820 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:29.964850 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:18:29.965497 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:29.965513 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:29.965862 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:29.966106 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:29.966462 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.967309 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.968468 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:29.968863 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:29.968968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:29.969725 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:29.969729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:29.969738 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:29.970118 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:29.970166 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:29.970184 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:29.970251 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:29.970402 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 20:18:29.970640 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20260620 20:18:29.971772 8385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:29.972597 8385 log.cc:824] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:29.973802 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:18:29.974408 8385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338: No bootstrap required, opened a new log
I20260620 20:18:29.977680 8385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c840a2a5444f9db4816afbe9ac0338" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34515 } }
I20260620 20:18:29.977854 8385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:29.977881 8385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c840a2a5444f9db4816afbe9ac0338, State: Initialized, Role: FOLLOWER
I20260620 20:18:29.978166 8385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c840a2a5444f9db4816afbe9ac0338" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34515 } }
I20260620 20:18:29.978253 8385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:29.978291 8385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:29.978345 8385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:29.978987 8385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c840a2a5444f9db4816afbe9ac0338" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34515 } }
I20260620 20:18:29.979264 8385 leader_election.cc:302] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73c840a2a5444f9db4816afbe9ac0338; no voters:
I20260620 20:18:29.979517 8385 leader_election.cc:288] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:29.979621 8390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:29.979820 8390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [term 1 LEADER]: Becoming Leader. State: Replica: 73c840a2a5444f9db4816afbe9ac0338, State: Running, Role: LEADER
I20260620 20:18:29.980147 8390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c840a2a5444f9db4816afbe9ac0338" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34515 } }
I20260620 20:18:29.980628 8385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:29.981660 8392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73c840a2a5444f9db4816afbe9ac0338. Latest consensus state: current_term: 1 leader_uuid: "73c840a2a5444f9db4816afbe9ac0338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c840a2a5444f9db4816afbe9ac0338" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34515 } } }
I20260620 20:18:29.981784 8392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:29.982117 8391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73c840a2a5444f9db4816afbe9ac0338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c840a2a5444f9db4816afbe9ac0338" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34515 } } }
I20260620 20:18:29.982247 8391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:29.982168 8399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:29.984997 8399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:29.989450 8399 catalog_manager.cc:1347] Generated new cluster ID: 07596dac06dc4d9a97ac8fdeba93fe43
I20260620 20:18:29.989501 8399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:30.002539 8399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:30.003511 8399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:30.011413 8399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73c840a2a5444f9db4816afbe9ac0338: Generated new TSK 0
I20260620 20:18:30.011986 8399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:30.027319 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:30.023005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.2.235.254:34515)
0620 20:18:30.023115 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:30.023119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:30.023301 (+ 182us) server_negotiation.cc:407] Connection header received
0620 20:18:30.023427 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:30.023429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:30.023464 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:30.023532 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:18:30.023740 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.024171 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.025322 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.025541 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.025616 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:30.025917 (+ 301us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:18:30.027045 (+ 1128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:18:30.027221 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:18:30.027374 8388 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 20:18:30.028920 8414 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:30.022960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34515 (local address 127.0.0.1:42108)
0620 20:18:30.023218 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:30.023231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:30.023331 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:30.023553 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:30.023556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:30.023563 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:18:30.023656 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:30.023661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.024261 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.024264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:30.025158 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:30.025169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.025651 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.025654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:30.025742 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:30.025769 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:18:30.027181 (+ 1412us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:18:30.028773 (+ 1592us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.235.254:34515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41795/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":1434,"thread_start_us":95,"threads_started":1}
I20260620 20:18:30.029304 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8318
E20260620 20:18:30.037403 8415 webserver.cc:533] Webserver: sslize failed:
E20260620 20:18:30.037469 8415 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T20:18:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (268 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T20:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:30Z Disabled control of system clock
I20260620 20:18:30.074914 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:36879
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:35543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:36879
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/testchainca.cert with env {}
W20260620 20:18:30.173118 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:30.173310 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:30.173342 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:30.176906 8419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:30.176946 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:30.176959 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:30.176972 8419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:30.176985 8419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:18:30.177032 8419 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:18:30.177052 8419 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 20:18:30.181519 8419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:36879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:30.182301 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:30.183074 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:30.190223 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:30.190230 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:30.190233 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:30.190459 8419 server_base.cc:1034] running on GCE node
I20260620 20:18:30.190975 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:30.191838 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:30.193007 8419 hybrid_clock.cc:648] HybridClock initialized: now 1781986710192996 us; error 30 us; skew 500 ppm
I20260620 20:18:30.194512 8419 webserver.cc:460] Webserver started at http://127.2.235.254:35471/ using document root <none> and password file <none>
I20260620 20:18:30.194962 8419 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:30.195039 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:30.195222 8419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:30.197024 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "c540a6a46958416885a3651fa5e83ab5"
format_stamp: "Formatted at 2026-06-20 20:18:30 on dist-test-slave-8z53"
server_key: "2131579952c843f95cf2ef59689ec0f7"
server_key_iv: "7903908dcd0efc2a675aa6ba5751d37f"
server_key_version: "encryptionkey@0"
I20260620 20:18:30.197561 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "c540a6a46958416885a3651fa5e83ab5"
format_stamp: "Formatted at 2026-06-20 20:18:30 on dist-test-slave-8z53"
server_key: "2131579952c843f95cf2ef59689ec0f7"
server_key_iv: "7903908dcd0efc2a675aa6ba5751d37f"
server_key_version: "encryptionkey@0"
I20260620 20:18:30.200557 8419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:18:30.202677 8433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:30.203670 8419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:30.203787 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "c540a6a46958416885a3651fa5e83ab5"
format_stamp: "Formatted at 2026-06-20 20:18:30 on dist-test-slave-8z53"
server_key: "2131579952c843f95cf2ef59689ec0f7"
server_key_iv: "7903908dcd0efc2a675aa6ba5751d37f"
server_key_version: "encryptionkey@0"
I20260620 20:18:30.203896 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:30.224824 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:30.225407 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:30.225580 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:30.231446 8419 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:36879
I20260620 20:18:30.231455 8485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:36879 every 8 connection(s)
I20260620 20:18:30.232177 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:30.234776 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:30.239353 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5: Bootstrap starting.
I20260620 20:18:30.239686 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8419
I20260620 20:18:30.239799 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:30.240067 2991 external_mini_cluster.cc:1428] Setting key 0b1b7db378e269d376d8c57342b4eadd
I20260620 20:18:30.242316 8486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:30.243007 8486 log.cc:824] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:30.244702 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5: No bootstrap required, opened a new log
I20260620 20:18:30.248112 8486 raft_consensus.cc:348] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540a6a46958416885a3651fa5e83ab5" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 36879 } }
I20260620 20:18:30.248253 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:30.241060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.2.235.254:36879)
0620 20:18:30.241665 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:30.241685 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:30.241727 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:18:30.242289 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:30.242303 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:30.242692 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:30.243022 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:30.243443 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.244466 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.245425 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.245856 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.246027 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:30.246822 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:30.246827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:30.246838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:30.247173 (+ 335us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:30.247218 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:30.247234 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:30.247333 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:30.247471 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 20:18:30.247681 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":82,"threads_started":1}
I20260620 20:18:30.248317 8486 raft_consensus.cc:374] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:30.248390 8486 raft_consensus.cc:729] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c540a6a46958416885a3651fa5e83ab5, State: Initialized, Role: FOLLOWER
I20260620 20:18:30.248879 8486 consensus_queue.cc:260] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540a6a46958416885a3651fa5e83ab5" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 36879 } }
I20260620 20:18:30.249014 8486 raft_consensus.cc:388] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:30.249069 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:30.249152 8486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:30.249905 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540a6a46958416885a3651fa5e83ab5" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 36879 } }
I20260620 20:18:30.250207 8486 leader_election.cc:302] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c540a6a46958416885a3651fa5e83ab5; no voters:
I20260620 20:18:30.250521 8486 leader_election.cc:288] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:30.250573 8491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:30.250830 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:18:30.250845 8491 raft_consensus.cc:686] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [term 1 LEADER]: Becoming Leader. State: Replica: c540a6a46958416885a3651fa5e83ab5, State: Running, Role: LEADER
I20260620 20:18:30.251091 8491 consensus_queue.cc:237] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540a6a46958416885a3651fa5e83ab5" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 36879 } }
I20260620 20:18:30.251547 8486 sys_catalog.cc:564] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:30.252470 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c540a6a46958416885a3651fa5e83ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540a6a46958416885a3651fa5e83ab5" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 36879 } } }
I20260620 20:18:30.252584 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:30.252699 8493 sys_catalog.cc:455] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c540a6a46958416885a3651fa5e83ab5. Latest consensus state: current_term: 1 leader_uuid: "c540a6a46958416885a3651fa5e83ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540a6a46958416885a3651fa5e83ab5" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 36879 } } }
I20260620 20:18:30.252777 8493 sys_catalog.cc:458] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:30.253007 8500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:30.255976 8500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:30.260355 8500 catalog_manager.cc:1347] Generated new cluster ID: 6eee446328ae4d03bf2392cc39fb3596
I20260620 20:18:30.260406 8500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:30.268137 8500 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:30.268805 8500 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:30.273871 8500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c540a6a46958416885a3651fa5e83ab5: Generated new TSK 0
I20260620 20:18:30.274403 8500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:30.747368 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:30.282889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.2.235.254:36879)
0620 20:18:30.283012 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:30.283015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:30.283202 (+ 187us) server_negotiation.cc:407] Connection header received
0620 20:18:30.283318 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:30.283320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:30.283355 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:30.283424 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:18:30.283669 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.284140 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.285366 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.285572 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.285645 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:30.285940 (+ 295us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:18:30.747026 (+461086us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:18:30.747258 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 20:18:30.747458 8489 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 20:18:30.747574 8515 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:30.282843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:36879 (local address 127.0.0.1:44622)
0620 20:18:30.283120 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:30.283135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:30.283229 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:30.283448 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:30.283451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:30.283458 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:18:30.283556 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:30.283561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.284250 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.284253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:30.285203 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:30.285213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:30.285680 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:30.285684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:30.285776 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:30.285802 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:18:30.747228 (+461426us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:18:30.747402 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.235.254:36879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
I20260620 20:18:30.748104 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8419
2026-06-20T20:18:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (698 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (105 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (109 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:31 dist-test-slave-8z53 krb5kdc[8527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:31 dist-test-slave-8z53 krb5kdc[8527](info): set up 2 sockets
Jun 20 20:18:31 dist-test-slave-8z53 krb5kdc[8527](info): commencing operation
krb5kdc: starting...
W20260620 20:18:31.913654 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:31 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:32.018188 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:38937
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:32.116750 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:32.116947 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:32.116981 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:32.120584 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:32.120626 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:32.120643 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:32.120657 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:32.120697 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:32.125245 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:32.126107 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:32.126973 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:32.134653 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.134666 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.134626 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:32.135093 8540 server_base.cc:1034] running on GCE node
I20260620 20:18:32.135614 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:32.136557 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:32.137714 8540 hybrid_clock.cc:648] HybridClock initialized: now 1781986712137692 us; error 36 us; skew 500 ppm
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.140204 8540 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:32.140939 8540 webserver.cc:460] Webserver started at http://127.2.235.254:35887/ using document root <none> and password file <none>
I20260620 20:18:32.141441 8540 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:32.141494 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:32.141676 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:32.143508 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "a1cb38aa4bfe4107babbd43626dde4d2"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "b05cc64a363ebf5ce0ebb99a3856f14f"
server_key_iv: "b01e19098d35c6fb592a4787bd214393"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.143975 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "a1cb38aa4bfe4107babbd43626dde4d2"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "b05cc64a363ebf5ce0ebb99a3856f14f"
server_key_iv: "b01e19098d35c6fb592a4787bd214393"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.147457 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:32.149736 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.150772 8540 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:32.150864 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "a1cb38aa4bfe4107babbd43626dde4d2"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "b05cc64a363ebf5ce0ebb99a3856f14f"
server_key_iv: "b01e19098d35c6fb592a4787bd214393"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.150933 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:32.169216 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:32.171691 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:32.171830 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:32.177959 8540 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:38937
I20260620 20:18:32.177984 8607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:38937 every 8 connection(s)
I20260620 20:18:32.178757 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:32.181049 8608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:32.182801 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8540
I20260620 20:18:32.182906 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:32.183179 2991 external_mini_cluster.cc:1428] Setting key 9a76ec601c149576cac193b0127cdb65
I20260620 20:18:32.186348 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2: Bootstrap starting.
I20260620 20:18:32.188237 8608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:32.188961 8608 log.cc:824] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:32.190672 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2: No bootstrap required, opened a new log
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:32.193563 8608 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38937 } }
I20260620 20:18:32.193768 8608 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:32.193835 8608 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cb38aa4bfe4107babbd43626dde4d2, State: Initialized, Role: FOLLOWER
I20260620 20:18:32.194253 8608 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38937 } }
I20260620 20:18:32.194383 8608 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:32.194450 8608 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:32.194561 8608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:32.195248 8608 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38937 } }
I20260620 20:18:32.195608 8608 leader_election.cc:302] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cb38aa4bfe4107babbd43626dde4d2; no voters:
I20260620 20:18:32.195888 8608 leader_election.cc:288] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:32.195978 8613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:32.196158 8613 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [term 1 LEADER]: Becoming Leader. State: Replica: a1cb38aa4bfe4107babbd43626dde4d2, State: Running, Role: LEADER
I20260620 20:18:32.196394 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.184418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.2.235.254:38937)
0620 20:18:32.184805 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.184831 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.184863 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:18:32.185644 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.185663 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.185954 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.186252 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.187038 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.187887 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.188714 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.189205 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.189416 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.191620 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.191625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.191634 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.191656 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.193658 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.194102 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.194106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.194110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.194199 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.194431 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.194434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.194435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.194831 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.195001 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.195228 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.195464 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260620 20:18:32.196664 8613 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38937 } }
I20260620 20:18:32.197044 8608 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:32.198947 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cb38aa4bfe4107babbd43626dde4d2. Latest consensus state: current_term: 1 leader_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38937 } } }
I20260620 20:18:32.199378 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:32.199571 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cb38aa4bfe4107babbd43626dde4d2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 38937 } } }
I20260620 20:18:32.199659 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2 [sys.catalog]: This master's current role is: LEADER
W20260620 20:18:32.201646 8627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:32.201723 8627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:32.201797 8629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:32.202392 8629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:32.206794 8629 catalog_manager.cc:1347] Generated new cluster ID: 323b9bf6dd434801838a4b8421f607b9
I20260620 20:18:32.206863 8629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:32.220461 8629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:32.221166 8629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:32.233172 8629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cb38aa4bfe4107babbd43626dde4d2: Generated new TSK 0
I20260620 20:18:32.233698 8629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:32.261607 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:38937
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:32.361382 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:32.361574 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:32.361610 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:32.365669 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:32.365722 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:32.365777 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:32.370357 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:32.371156 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:32.371932 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:32.380125 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.380183 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.380183 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:32.380290 8634 server_base.cc:1034] running on GCE node
I20260620 20:18:32.380795 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:32.381418 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:32.382567 8634 hybrid_clock.cc:648] HybridClock initialized: now 1781986712382555 us; error 23 us; skew 500 ppm
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.385018 8634 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:32.385905 8634 webserver.cc:460] Webserver started at http://127.2.235.193:34911/ using document root <none> and password file <none>
I20260620 20:18:32.386425 8634 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:32.386518 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:32.386745 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:32.388489 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "fafd64f9d3ac439386e86516590955ce"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "9b4c49910d919d7d0cb2bcc5ce89882b"
server_key_iv: "669b391130cc008042d03ba305239ed7"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.388990 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "fafd64f9d3ac439386e86516590955ce"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "9b4c49910d919d7d0cb2bcc5ce89882b"
server_key_iv: "669b391130cc008042d03ba305239ed7"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.392311 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:18:32.394425 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.395407 8634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:32.395560 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "fafd64f9d3ac439386e86516590955ce"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "9b4c49910d919d7d0cb2bcc5ce89882b"
server_key_iv: "669b391130cc008042d03ba305239ed7"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.395666 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:32.411275 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:32.414028 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:32.414193 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:32.414655 8634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:32.415346 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:32.415411 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.415478 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:32.415537 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.423265 8634 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:42945
I20260620 20:18:32.423280 8762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:42945 every 8 connection(s)
I20260620 20:18:32.424057 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:32.426221 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8634
I20260620 20:18:32.426344 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:32.426656 2991 external_mini_cluster.cc:1428] Setting key b16663bb27bbb757269896efe4a3a201
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:32.438046 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.425634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:41573 (local address 127.2.235.254:38937)
0620 20:18:32.425755 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.425759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.426575 (+ 816us) server_negotiation.cc:407] Connection header received
0620 20:18:32.428061 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.428065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.428106 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.428192 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.429328 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.430051 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.431486 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.431691 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.431787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.434346 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.434350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.434352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.434379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.436096 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.436630 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.436634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.436636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.436742 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.437076 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.437078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.437079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.437229 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.437373 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.437847 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.437941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:18:32.438843 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.425982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:38937 (local address 127.2.235.193:41573)
0620 20:18:32.426421 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:32.426470 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:32.427861 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:32.428347 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:32.428353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:32.428766 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:32.429165 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:32.429177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.430182 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.430187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:32.431377 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:32.431386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.431829 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.431832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:32.431996 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.432542 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:32.432557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:32.434183 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:32.436211 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:32.436216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:32.436226 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:32.436548 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:32.436854 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:32.436857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:32.436859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:32.436976 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:32.437412 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:32.437420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:32.437633 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 20:18:32.437780 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:18:32.437985 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
I20260620 20:18:32.439911 8763 heartbeater.cc:347] Connected to a master server at 127.2.235.254:38937
I20260620 20:18:32.440114 8763 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:32.440577 8763 heartbeater.cc:511] Master 127.2.235.254:38937 requested a full tablet report, sending...
I20260620 20:18:32.444681 8572 ts_manager.cc:194] Registered new tserver with Master: fafd64f9d3ac439386e86516590955ce (127.2.235.193:42945)
I20260620 20:18:32.446530 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:41573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:32.452924 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:38937
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:32.552795 8768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:32.553030 8768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:32.553097 8768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:32.557346 8768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:32.557415 8768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:32.557516 8768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:32.562427 8768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:32.563259 8768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:32.564141 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:32.572455 8773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.572448 8774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.572448 8776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:32.572810 8768 server_base.cc:1034] running on GCE node
I20260620 20:18:32.573143 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:32.573707 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:32.574869 8768 hybrid_clock.cc:648] HybridClock initialized: now 1781986712574846 us; error 38 us; skew 500 ppm
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.577415 8768 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:32.578212 8768 webserver.cc:460] Webserver started at http://127.2.235.194:37685/ using document root <none> and password file <none>
I20260620 20:18:32.578704 8768 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:32.578783 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:32.578963 8768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:32.580615 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "280b6b6090c14aaeae2dc4461783fa22"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "03a0d8916dae9469683a90a1c7179b41"
server_key_iv: "9de2923afa060e27719645f17836fa4c"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.581176 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "280b6b6090c14aaeae2dc4461783fa22"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "03a0d8916dae9469683a90a1c7179b41"
server_key_iv: "9de2923afa060e27719645f17836fa4c"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.584493 8768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:32.586879 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.587972 8768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:32.588084 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "280b6b6090c14aaeae2dc4461783fa22"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "03a0d8916dae9469683a90a1c7179b41"
server_key_iv: "9de2923afa060e27719645f17836fa4c"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.588173 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:32.611258 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:32.614166 8768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:32.614348 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:32.614850 8768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:32.615648 8768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:32.615716 8768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.615784 8768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:32.615857 8768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.625226 8768 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:33441
I20260620 20:18:32.625305 8896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:33441 every 8 connection(s)
I20260620 20:18:32.626072 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:32.630373 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8768
I20260620 20:18:32.630493 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:32.630726 2991 external_mini_cluster.cc:1428] Setting key 298af2bb4784be434210ba8bed3db16b
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:32.638963 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.627756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:52483 (local address 127.2.235.254:38937)
0620 20:18:32.627889 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.627893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.628610 (+ 717us) server_negotiation.cc:407] Connection header received
0620 20:18:32.629918 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.629920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.629960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.630023 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.631088 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.631667 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.632781 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.633125 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.633231 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.635680 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.635683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.635685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.635708 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.637085 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.637574 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.637579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.637580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.637653 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.638009 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.638012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.638014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.638170 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.638265 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.638692 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.638794 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:18:32.639705 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.628105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:38937 (local address 127.2.235.194:52483)
0620 20:18:32.628480 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:32.628514 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:32.629735 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:32.630161 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:32.630167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:32.630477 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:32.630956 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:32.630969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.631784 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.631787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:32.632669 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:32.632677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.633244 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.633283 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:32.633491 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.633988 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:32.634010 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:32.635502 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:32.637206 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:32.637211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:32.637222 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:32.637479 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:32.637759 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:32.637763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:32.637765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:32.637899 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:32.638265 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:32.638271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:32.638475 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 20:18:32.638630 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:18:32.638927 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
I20260620 20:18:32.640689 8897 heartbeater.cc:347] Connected to a master server at 127.2.235.254:38937
I20260620 20:18:32.640949 8897 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:18:32.641502 8897 heartbeater.cc:511] Master 127.2.235.254:38937 requested a full tablet report, sending...
I20260620 20:18:32.642418 8572 ts_manager.cc:194] Registered new tserver with Master: 280b6b6090c14aaeae2dc4461783fa22 (127.2.235.194:33441)
I20260620 20:18:32.642971 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:52483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:32.654281 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:38937
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:32.758708 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:32.758919 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:32.758960 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:32.762543 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:32.762588 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:32.762639 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:32.768584 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:32.769560 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:32.770372 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:32.778668 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:32.778810 8902 server_base.cc:1034] running on GCE node
W20260620 20:18:32.778668 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:32.778720 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:32.779232 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:32.779784 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:32.780977 8902 hybrid_clock.cc:648] HybridClock initialized: now 1781986712780962 us; error 32 us; skew 500 ppm
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.783676 8902 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:32.784348 8902 webserver.cc:460] Webserver started at http://127.2.235.195:33229/ using document root <none> and password file <none>
I20260620 20:18:32.784778 8902 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:32.784834 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:32.784973 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:32.786810 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "0aa4e9b5e04f4fa38ca767518a5dd9b1"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "b72416ed8a3d2672ee1c0cd3477e50c2"
server_key_iv: "c411b223c324445d4ee4357c57f0bedb"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.787256 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "0aa4e9b5e04f4fa38ca767518a5dd9b1"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "b72416ed8a3d2672ee1c0cd3477e50c2"
server_key_iv: "c411b223c324445d4ee4357c57f0bedb"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.790535 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:18:32.792794 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.793877 8902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:32.793964 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "0aa4e9b5e04f4fa38ca767518a5dd9b1"
format_stamp: "Formatted at 2026-06-20 20:18:32 on dist-test-slave-8z53"
server_key: "b72416ed8a3d2672ee1c0cd3477e50c2"
server_key_iv: "c411b223c324445d4ee4357c57f0bedb"
server_key_version: "encryptionkey@0"
I20260620 20:18:32.794036 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:32.804322 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:32.807205 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:32.807339 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:32.807783 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:32.808441 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:32.808485 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.808521 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:32.808537 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:32.816553 8902 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:42367
I20260620 20:18:32.816628 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:42367 every 8 connection(s)
I20260620 20:18:32.817360 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:32.819334 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 8902
I20260620 20:18:32.819447 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:32.819766 2991 external_mini_cluster.cc:1428] Setting key 9d0e3cc7a0170c58c43626f96d547ae8
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:32.829986 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.819029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:52113 (local address 127.2.235.254:38937)
0620 20:18:32.819207 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.819212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.819901 (+ 689us) server_negotiation.cc:407] Connection header received
0620 20:18:32.821308 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.821312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.821365 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.821473 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.822486 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.823020 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.824154 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.824375 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.824480 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.826819 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.826822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.826824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.826844 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.828186 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.828665 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.828668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.828669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.828740 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.829042 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.829044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.829045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.829219 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.829374 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.829791 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.829879 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:18:32.830691 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.819267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:38937 (local address 127.2.235.195:52113)
0620 20:18:32.819752 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:32.819814 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:32.821078 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:32.821633 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:32.821639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:32.821942 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:32.822354 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:32.822365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.823139 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.823142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:32.824012 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:32.824021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.824510 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.824515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:32.824669 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.825167 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:32.825182 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:32.826658 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:32.828296 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:32.828300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:32.828309 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:32.828572 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:32.828824 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:32.828826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:32.828827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:32.828951 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:32.829390 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:32.829396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:32.829596 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 20:18:32.829748 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 20:18:32.829968 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":154,"threads_started":1}
I20260620 20:18:32.831693 9031 heartbeater.cc:347] Connected to a master server at 127.2.235.254:38937
I20260620 20:18:32.831930 9031 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:32.832422 9031 heartbeater.cc:511] Master 127.2.235.254:38937 requested a full tablet report, sending...
I20260620 20:18:32.833539 8571 ts_manager.cc:194] Registered new tserver with Master: 0aa4e9b5e04f4fa38ca767518a5dd9b1 (127.2.235.195:42367)
I20260620 20:18:32.834103 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:52113
I20260620 20:18:32.845144 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/test-admin.keytab.
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.860592 2991 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 20:18:32.863161 9038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:32.861902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.2.235.193:42945)
0620 20:18:32.862199 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.862207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.862227 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:18:32.862268 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.862272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.862453 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.862559 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.862995 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40434: BlockingRecv error: recv got EOF from 127.0.0.1:40434 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":118,"threads_started":1}
W20260620 20:18:32.863433 2991 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.864372 2991 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.865231 2991 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:32.874159 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.866112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.2.235.193:42945)
0620 20:18:32.866236 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.866241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.866356 (+ 115us) server_negotiation.cc:407] Connection header received
0620 20:18:32.866504 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.866506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.866542 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.866622 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.867170 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.867935 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.868759 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.869085 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.869196 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.871269 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.871276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.871286 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.871306 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.872605 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.873024 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.873028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.873029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.873098 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.873390 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.873392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.873393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.873588 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.873724 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.873875 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.874008 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 20:18:32.877444 9038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:32.876478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.2.235.193:42945)
0620 20:18:32.876593 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.876596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.876741 (+ 145us) server_negotiation.cc:407] Connection header received
0620 20:18:32.876856 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.876859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.876895 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.876967 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.877347 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40454: BlockingRecv error: recv got EOF from 127.0.0.1:40454 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 20:18:32.877722 2991 init.cc:244] Error while opening credential cache '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc)
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.878604 2991 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.879467 2991 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:32.887189 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.880220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.2.235.193:42945)
0620 20:18:32.880343 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.880346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.880358 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:32.880460 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.880462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.880505 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.880562 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.880979 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.881501 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.882216 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.882426 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.882497 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.884553 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.884556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.884558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.884577 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.885882 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.886235 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.886237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.886239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.886300 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.886534 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.886536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.886537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.886667 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.886763 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.886943 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.887046 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 20:18:32.889355 9038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:32.888616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.2.235.193:42945)
0620 20:18:32.888727 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.888730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.888860 (+ 130us) server_negotiation.cc:407] Connection header received
0620 20:18:32.888920 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.888921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.888947 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.889000 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.889240 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40468: BlockingRecv error: recv got EOF from 127.0.0.1:40468 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 20:18:32.889469 2991 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986686246774-2991-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.890342 2991 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:32.891099 2991 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:18:32.898500 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:32.891759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.2.235.193:42945)
0620 20:18:32.891884 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:32.891887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:32.891897 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:18:32.891975 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:32.891977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:32.892003 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:32.892062 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:32.892485 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.892944 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.893782 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:32.893987 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:32.894053 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:32.895835 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:32.895837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:32.895839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:32.895858 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:32.897191 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.897615 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.897617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.897618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.897676 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:32.897931 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:32.897933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:32.897934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:32.898081 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:32.898167 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:32.898331 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:18:32.898399 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:18:32.899454 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8634
I20260620 20:18:32.904953 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8768
I20260620 20:18:32.909874 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8902
I20260620 20:18:32.916692 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 8540
2026-06-20T20:18:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1956 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[9051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[9051](info): set up 2 sockets
Jun 20 20:18:32 dist-test-slave-8z53 krb5kdc[9051](info): commencing operation
krb5kdc: starting...
W20260620 20:18:33.889645 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:33 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab.
Entry for principal oryx/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab.
I20260620 20:18:33.995831 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34099
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab
--principal=oryx/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:34099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:34.090464 9064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:34.090649 9064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:34.090682 9064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:34.094187 9064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:34.094238 9064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:34.094255 9064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:34.094269 9064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:34.094283 9064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:34.098728 9064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:34099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab
--principal=oryx/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:34.099459 9064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:34.100198 9064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:34.107566 9070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.107668 9064 server_base.cc:1034] running on GCE node
W20260620 20:18:34.107597 9069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.107610 9072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.108309 9064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:34.109302 9064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:34.110483 9064 hybrid_clock.cc:648] HybridClock initialized: now 1781986714110462 us; error 35 us; skew 500 ppm
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:34.112857 9064 init.cc:377] Logged in from keytab as oryx/127.2.235.254@KRBTEST.COM (short username oryx)
I20260620 20:18:34.113618 9064 webserver.cc:460] Webserver started at http://127.2.235.254:45987/ using document root <none> and password file <none>
I20260620 20:18:34.114081 9064 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:34.114164 9064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:34.114365 9064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:34.116091 9064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "33ff3378d82e43a0894307e92d2a1997"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "f24af6da488ee9c24480af6b58086e8a"
server_key_iv: "63b03177e38e9c01cb145c705d2d1868"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.116622 9064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "33ff3378d82e43a0894307e92d2a1997"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "f24af6da488ee9c24480af6b58086e8a"
server_key_iv: "63b03177e38e9c01cb145c705d2d1868"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.119743 9064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:34.121790 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.122754 9064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:34.122879 9064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "33ff3378d82e43a0894307e92d2a1997"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "f24af6da488ee9c24480af6b58086e8a"
server_key_iv: "63b03177e38e9c01cb145c705d2d1868"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.122969 9064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:34.133941 9064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:34.136248 9064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:34.136428 9064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:34.142861 9064 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:34099
I20260620 20:18:34.143131 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:34099 every 8 connection(s)
I20260620 20:18:34.143769 9064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:34.146549 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:34.150235 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9064
I20260620 20:18:34.150384 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:34.150640 2991 external_mini_cluster.cc:1428] Setting key d860dcf062a4c3e86eaa8541722244a0
I20260620 20:18:34.151585 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Bootstrap starting.
I20260620 20:18:34.153489 9132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:34.154258 9132 log.cc:824] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:34.156055 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: No bootstrap required, opened a new log
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:34.159849 9132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } }
I20260620 20:18:34.160043 9132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:34.160099 9132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ff3378d82e43a0894307e92d2a1997, State: Initialized, Role: FOLLOWER
I20260620 20:18:34.160455 9132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } }
I20260620 20:18:34.160593 9132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:34.160677 9132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:34.160785 9132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:34.161526 9132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } }
I20260620 20:18:34.161806 9132 leader_election.cc:302] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33ff3378d82e43a0894307e92d2a1997; no voters:
I20260620 20:18:34.162106 9132 leader_election.cc:288] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:34.162168 9137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:34.162346 9137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 LEADER]: Becoming Leader. State: Replica: 33ff3378d82e43a0894307e92d2a1997, State: Running, Role: LEADER
I20260620 20:18:34.162652 9137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } }
I20260620 20:18:34.163190 9132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:34.164091 9138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } }
I20260620 20:18:34.164206 9138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:34.164376 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.151842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.2.235.254:34099)
0620 20:18:34.152182 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:34.152198 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:34.152236 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:18:34.152762 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:34.152778 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:34.153040 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:34.153367 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:34.154113 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.155000 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.156066 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.156556 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.156667 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.159153 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:34.159158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:34.159168 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:34.159190 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:34.161027 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.161477 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.161482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.161487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.161593 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.161926 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.161929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.161931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.162358 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:34.163082 (+ 724us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:34.163186 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 20:18:34.163448 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
I20260620 20:18:34.164582 9146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:34.163703 9137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ff3378d82e43a0894307e92d2a1997. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } }
I20260620 20:18:34.164829 9137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:34.167445 9146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:34.172248 9146 catalog_manager.cc:1347] Generated new cluster ID: 6a746e6d0f4341209871a0b8182d8522
I20260620 20:18:34.172320 9146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:34.192169 9146 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:34.192945 9146 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:34.198081 9146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Generated new TSK 0
I20260620 20:18:34.198719 9146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab.
Entry for principal oryx/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab.
I20260620 20:18:34.227707 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34099
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:34.327193 9158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:34.327389 9158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:34.327425 9158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:34.331199 9158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:34.331246 9158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:34.331306 9158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:34.336041 9158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:34.336875 9158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:34.337812 9158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:34.346413 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.346509 9158 server_base.cc:1034] running on GCE node
W20260620 20:18:34.346401 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.346401 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.347101 9158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:34.347646 9158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:34.348845 9158 hybrid_clock.cc:648] HybridClock initialized: now 1781986714348816 us; error 40 us; skew 500 ppm
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:34.351399 9158 init.cc:377] Logged in from keytab as oryx/127.2.235.193@KRBTEST.COM (short username oryx)
I20260620 20:18:34.352146 9158 webserver.cc:460] Webserver started at http://127.2.235.193:33017/ using document root <none> and password file <none>
I20260620 20:18:34.352617 9158 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:34.352705 9158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:34.352921 9158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:34.354653 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "df947cfb30144ad88d70d12a298746f8"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "3465588d0e29478e3bef09dcd4d6760b"
server_key_iv: "17798b79a6565d70c0cb6a644d77c7c8"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.355151 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "df947cfb30144ad88d70d12a298746f8"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "3465588d0e29478e3bef09dcd4d6760b"
server_key_iv: "17798b79a6565d70c0cb6a644d77c7c8"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.358508 9158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:18:34.360581 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.361783 9158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:18:34.361917 9158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "df947cfb30144ad88d70d12a298746f8"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "3465588d0e29478e3bef09dcd4d6760b"
server_key_iv: "17798b79a6565d70c0cb6a644d77c7c8"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.362015 9158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:34.387789 9158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:34.390358 9158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:34.390544 9158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:34.391037 9158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:34.391741 9158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:34.391819 9158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.391881 9158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:34.391930 9158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.400348 9158 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40531
I20260620 20:18:34.400360 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40531 every 8 connection(s)
I20260620 20:18:34.401098 9158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:34.402400 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9158
I20260620 20:18:34.402524 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:34.402814 2991 external_mini_cluster.cc:1428] Setting key 1e4f72a724036da411c523f6fefc5c21
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.194@KRBTEST.COM" created.
I20260620 20:18:34.414052 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.402642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:41789 (local address 127.2.235.254:34099)
0620 20:18:34.402773 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:34.402778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:34.403559 (+ 781us) server_negotiation.cc:407] Connection header received
0620 20:18:34.405126 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:34.405129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:34.405166 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:34.405226 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:34.406513 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.407034 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.408094 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.408366 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.408468 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.410681 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:34.410684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:34.410687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:34.410710 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:34.412102 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.412511 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.412513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.412514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.412576 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.412931 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.412936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.412937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.413177 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:34.413320 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:34.413849 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 20:18:34.413942 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:18:34.415017 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.402976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:41789)
0620 20:18:34.403438 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:34.403479 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:34.404919 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:34.405437 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:34.405449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:34.405941 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:34.406373 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:34.406386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.407145 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.407148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:34.407978 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:34.407987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.408512 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.408517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:34.408667 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.409194 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:34.409209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:34.410521 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:34.412206 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:34.412210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:34.412219 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:34.412436 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:34.412677 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:34.412680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:34.412682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:34.412834 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:34.413337 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:34.413360 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:34.413606 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 20:18:34.413809 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 20:18:34.414139 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260620 20:18:34.416106 9287 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34099
I20260620 20:18:34.416323 9287 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:34.416749 9287 heartbeater.cc:511] Master 127.2.235.254:34099 requested a full tablet report, sending...
I20260620 20:18:34.418113 9096 ts_manager.cc:194] Registered new tserver with Master: df947cfb30144ad88d70d12a298746f8 (127.2.235.193:40531)
I20260620 20:18:34.419199 9096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:41789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab.
Entry for principal oryx/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab.
I20260620 20:18:34.426247 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34099
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:34.531373 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:34.531579 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:34.531618 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:34.535519 9292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:34.535569 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:34.535622 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:34.540674 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:34.541605 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:34.542512 9292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:34.551057 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.551081 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.551070 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.551425 9292 server_base.cc:1034] running on GCE node
I20260620 20:18:34.551827 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:34.552459 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:34.553710 9292 hybrid_clock.cc:648] HybridClock initialized: now 1781986714553702 us; error 32 us; skew 500 ppm
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:34.556280 9292 init.cc:377] Logged in from keytab as oryx/127.2.235.194@KRBTEST.COM (short username oryx)
I20260620 20:18:34.557099 9292 webserver.cc:460] Webserver started at http://127.2.235.194:34687/ using document root <none> and password file <none>
I20260620 20:18:34.557718 9292 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:34.557821 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:34.558017 9292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:34.559806 9292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "a7599a977b004257a29b1f305af05e8f"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "2a722efe77f17d6e2de95e1c84eeada8"
server_key_iv: "994325ec95eeebc4caa76772010016eb"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.560356 9292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "a7599a977b004257a29b1f305af05e8f"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "2a722efe77f17d6e2de95e1c84eeada8"
server_key_iv: "994325ec95eeebc4caa76772010016eb"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.563792 9292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:18:34.566155 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.567291 9292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:34.567425 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "a7599a977b004257a29b1f305af05e8f"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "2a722efe77f17d6e2de95e1c84eeada8"
server_key_iv: "994325ec95eeebc4caa76772010016eb"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.567525 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:34.583175 9292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:34.585696 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:34.585883 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:34.586390 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:34.587124 9292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:34.587190 9292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.587252 9292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:34.587304 9292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.595875 9292 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:39229
I20260620 20:18:34.595908 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:39229 every 8 connection(s)
I20260620 20:18:34.596655 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:34.601691 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9292
I20260620 20:18:34.601938 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:34.602288 2991 external_mini_cluster.cc:1428] Setting key 005804d45ddb574407c37436aec48782
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:34.611223 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.598304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:48743 (local address 127.2.235.254:34099)
0620 20:18:34.598445 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:34.598450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:34.599268 (+ 818us) server_negotiation.cc:407] Connection header received
0620 20:18:34.600437 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:34.600440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:34.600485 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:34.600552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:34.601943 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.602739 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.604060 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.604295 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.604375 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.607215 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:34.607222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:34.607228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:34.607261 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:34.609002 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.609655 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.609661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.609665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.609760 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.610113 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.610116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.610119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.610340 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:34.610456 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:34.610889 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 20:18:34.611021 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:18:34.611864 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.598668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.194:48743)
0620 20:18:34.599133 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:34.599171 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:34.600252 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:34.600669 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:34.600675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:34.601160 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:34.601678 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:34.601692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.602900 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.602903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:34.603880 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:34.603892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.604452 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.604459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:34.604697 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.605315 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:34.605344 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:34.607046 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:34.609171 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:34.609175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:34.609185 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:34.609530 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:34.609873 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:34.609876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:34.609877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:34.610006 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:34.610443 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:34.610447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:34.610640 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 20:18:34.610786 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 20:18:34.611004 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for oryx/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:18:34.613134 9421 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34099
I20260620 20:18:34.613389 9421 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:34.613863 9421 heartbeater.cc:511] Master 127.2.235.254:34099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.195@KRBTEST.COM" created.
I20260620 20:18:34.614980 9096 ts_manager.cc:194] Registered new tserver with Master: a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:34.615619 9096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:48743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab.
Entry for principal oryx/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab.
I20260620 20:18:34.627777 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34099
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:34.728057 9426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:34.728296 9426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:34.728360 9426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:34.732039 9426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:34.732118 9426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:34.732197 9426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:34.736864 9426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:34099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:34.737819 9426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:34.738670 9426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:34.746745 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.746739 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.746990 9426 server_base.cc:1034] running on GCE node
W20260620 20:18:34.746740 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.747430 9426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:34.747999 9426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:34.749174 9426 hybrid_clock.cc:648] HybridClock initialized: now 1781986714749162 us; error 24 us; skew 500 ppm
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:34.751652 9426 init.cc:377] Logged in from keytab as oryx/127.2.235.195@KRBTEST.COM (short username oryx)
I20260620 20:18:34.752319 9426 webserver.cc:460] Webserver started at http://127.2.235.195:41551/ using document root <none> and password file <none>
I20260620 20:18:34.752749 9426 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:34.752823 9426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:34.752961 9426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:34.754786 9426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "8b39446a91e14c1e8c0c01e650194f95"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "d0335a29ef64a4eedb5169b1b85b5846"
server_key_iv: "8554f2ba292e004a4d1363996341b345"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.755264 9426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "8b39446a91e14c1e8c0c01e650194f95"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "d0335a29ef64a4eedb5169b1b85b5846"
server_key_iv: "8554f2ba292e004a4d1363996341b345"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.758579 9426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:18:34.760566 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.761723 9426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:34.761826 9426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "8b39446a91e14c1e8c0c01e650194f95"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "d0335a29ef64a4eedb5169b1b85b5846"
server_key_iv: "8554f2ba292e004a4d1363996341b345"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.761931 9426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:34.784255 9426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:34.786849 9426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:34.787034 9426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:34.787533 9426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:34.788249 9426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:34.788291 9426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.788367 9426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:34.788401 9426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.796566 9426 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:40047
I20260620 20:18:34.796593 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:40047 every 8 connection(s)
I20260620 20:18:34.797365 9426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:34.802603 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9426
I20260620 20:18:34.802704 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:34.802953 2991 external_mini_cluster.cc:1428] Setting key fa197003c54e8ec4f17b439b9271726c
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:34.810014 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.798950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:60095 (local address 127.2.235.254:34099)
0620 20:18:34.799070 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:34.799073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:34.800113 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 20:18:34.800891 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:34.800895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:34.800950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:34.801041 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:34.802058 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.802674 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.804043 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.804331 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.804408 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.806888 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:34.806891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:34.806893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:34.806915 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:34.808203 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.808651 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.808653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.808654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.808732 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:34.809057 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:34.809059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:34.809061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:34.809227 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:34.809381 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:34.809811 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 20:18:34.809901 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:18:34.810740 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:34.799248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:60095)
0620 20:18:34.799992 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:34.800027 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:34.800703 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:34.801186 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:34.801191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:34.801538 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:34.801931 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:34.801942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.802786 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.802792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:34.803923 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:34.803932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:34.804466 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:34.804469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:34.804642 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:34.805336 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:34.805351 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:34.806742 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:34.808327 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:34.808331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:34.808341 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:34.808565 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:34.808836 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:34.808838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:34.808839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:34.808976 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:34.809392 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:34.809397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:34.809588 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 20:18:34.809729 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 20:18:34.809959 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":464,"threads_started":1}
I20260620 20:18:34.811621 9555 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34099
I20260620 20:18:34.811861 9555 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:34.812266 9555 heartbeater.cc:511] Master 127.2.235.254:34099 requested a full tablet report, sending...
I20260620 20:18:34.813128 9096 ts_manager.cc:194] Registered new tserver with Master: 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047)
I20260620 20:18:34.813673 9096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:60095
I20260620 20:18:34.816517 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.253.keytab.
Entry for principal oryx/127.2.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.253.keytab.
I20260620 20:18:34.838397 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:39357
--webserver_interface=127.2.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.253.keytab
--principal=oryx/127.2.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:34099,127.2.235.253:39357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:34.940856 9560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:34.941059 9560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:34.941093 9560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:34.944738 9560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:34.944782 9560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:34.944799 9560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:34.944814 9560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:34.944828 9560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:34.949643 9560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:34099,127.2.235.253:39357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.253.keytab
--principal=oryx/127.2.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
W20260620 20:18:34.949774 9560 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.235.254:34099,127.2.235.253:39357'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 20:18:34.950403 9560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:34.951165 9560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:34.959000 9568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.959017 9566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:34.959017 9565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:34.959164 9560 server_base.cc:1034] running on GCE node
I20260620 20:18:34.959782 9560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:34.960703 9560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:34.961844 9560 hybrid_clock.cc:648] HybridClock initialized: now 1781986714961820 us; error 38 us; skew 500 ppm
Jun 20 20:18:34 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:34.967422 9560 init.cc:377] Logged in from keytab as oryx/127.2.235.253@KRBTEST.COM (short username oryx)
I20260620 20:18:34.968204 9560 webserver.cc:460] Webserver started at http://127.2.235.253:43043/ using document root <none> and password file <none>
I20260620 20:18:34.968679 9560 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:34.968757 9560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:34.968963 9560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:34.970847 9560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/instance:
uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "130c9ea9f77dc5f2316918cad67573fd"
server_key_iv: "2bc1c8336550fc735b8a4716f5141496"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.971429 9560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal/instance:
uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "130c9ea9f77dc5f2316918cad67573fd"
server_key_iv: "2bc1c8336550fc735b8a4716f5141496"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.974740 9560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:34.976804 9575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:34.977947 9560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:34.978080 9560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "130c9ea9f77dc5f2316918cad67573fd"
server_key_iv: "2bc1c8336550fc735b8a4716f5141496"
server_key_version: "encryptionkey@0"
I20260620 20:18:34.978189 9560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:34.995688 9560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:34.998525 9560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:34.998696 9560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:35.004772 9560 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.253:39357
I20260620 20:18:35.005040 9627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.253:39357 every 8 connection(s)
I20260620 20:18:35.005654 9560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
I20260620 20:18:35.008096 9628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:35.011308 9628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:18:35.013320 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9560
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.253@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
I20260620 20:18:35.023411 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.012843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.2.235.254:34099)
0620 20:18:35.012959 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.012962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.013617 (+ 655us) server_negotiation.cc:407] Connection header received
0620 20:18:35.014581 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.014584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.014641 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.014738 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.015523 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.016077 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.017076 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.017347 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.017453 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.019768 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.019771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.019773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.019798 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.021355 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.021846 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.021849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.021851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.021933 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.022285 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.022288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.022289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.022579 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.022684 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.023126 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.023246 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:35.024005 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.014100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.2.235.253:39357)
0620 20:18:35.014440 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.014452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.014489 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:18:35.014624 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.014628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.014778 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.014945 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.015666 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.016530 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.017364 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.017747 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.017849 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.020240 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.020259 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.020271 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.020293 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.021853 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.022293 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.022298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.022300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.022364 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.022601 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.022624 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.022628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.022883 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.023009 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.023252 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.023362 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":251,"thread_start_us":142,"threads_started":1}
I20260620 20:18:35.024003 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.013062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:50606)
0620 20:18:35.013465 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.013507 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.014390 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.014791 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.014799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.015109 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.015396 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.015407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.016215 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.016218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.016950 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.016958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.017478 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.017482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.017685 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.018176 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.018196 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.019644 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.021458 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.021465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.021471 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.021751 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.022025 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.022029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.022033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.022194 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.022685 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.022690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.022892 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.023052 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.023304 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":24,"spinlock_wait_cycles":11776,"thread_start_us":112,"threads_started":1}
I20260620 20:18:35.025214 9628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.253@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
I20260620 20:18:35.031695 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.025615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:35168)
0620 20:18:35.025690 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.025702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.025831 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.026087 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.026091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.026227 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.026305 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.026310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.027025 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.027027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.027502 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.027507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.027854 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.027856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.027917 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.028219 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.028229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.029200 (+ 971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.030703 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.030705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.030706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.030872 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.031081 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.031083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.031084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.031143 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.031390 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.031392 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.031448 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.031513 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.031596 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 20:18:35.031764 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.025741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.2.235.253:39357)
0620 20:18:35.025878 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.025881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.025897 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:35.025946 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.025948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.025987 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.026047 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.026425 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.026915 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.027582 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.027775 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.027840 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.029342 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.029345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.029346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.029364 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.030619 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.030956 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.030958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.030959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.031016 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.031205 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.031207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.031208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.031325 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.031386 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.031559 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.031636 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:18:35.034211 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Bootstrap starting.
I20260620 20:18:35.036093 9628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:35.036758 9628 log.cc:824] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:35.038403 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: No bootstrap required, opened a new log
I20260620 20:18:35.040880 9628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } }
I20260620 20:18:35.041038 9628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:35.041116 9628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f79b50b6fdb46388c9e8d5cb38ba4fd, State: Initialized, Role: FOLLOWER
I20260620 20:18:35.041538 9628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } }
I20260620 20:18:35.041936 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } }
I20260620 20:18:35.042069 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:35.042522 9628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:18:35.046345 9645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:35.046432 9645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:35.048076 9560 master_runner.cc:428] Detected that this master 1f79b50b6fdb46388c9e8d5cb38ba4fd is joining an existing cluster
I20260620 20:18:35.048136 9560 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.235.253:39357
I20260620 20:18:35.057796 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.050013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:50618)
0620 20:18:35.050248 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.050263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.050385 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.050735 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.050738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.050937 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.051077 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.051081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.051989 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.051991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.052514 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.052521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.053032 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.053035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.053121 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.053552 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.053567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.053913 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.056194 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.056200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.056205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.056513 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.056840 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.056842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.056844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.056948 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.057310 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.057313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.057393 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.057500 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.057618 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":49,"thread_start_us":101,"threads_started":1}
I20260620 20:18:35.057804 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.049982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.2.235.254:34099)
0620 20:18:35.050099 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.050105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.050327 (+ 222us) server_negotiation.cc:407] Connection header received
0620 20:18:35.050498 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.050503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.050546 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.050596 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.051183 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.051876 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.052633 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.052923 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.053014 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.054075 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.054079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.054082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.054104 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.055840 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.056629 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.056634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.056637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.056736 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.057021 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.057024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.057025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.057177 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.057323 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.057559 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.057655 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:18:35.058951 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.050350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.2.235.253:39357)
0620 20:18:35.050431 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.050435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.050447 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:35.050520 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.050523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.050557 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.050635 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.052012 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.052691 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.053715 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.053914 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.054027 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.055232 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.055235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.055236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.055254 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.056859 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.057920 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.057924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.057926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.058002 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.058241 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.058243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.058245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.058392 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.058466 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.058727 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.058845 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":31}
I20260620 20:18:35.058961 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.050023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:35172)
0620 20:18:35.050365 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.050376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.050443 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.050666 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.050669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.050915 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.051074 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.051081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.052827 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.052829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.053600 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.053608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.054035 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.054038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.054151 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.054578 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.054596 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.054933 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.056962 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.056965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.056967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.057176 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.058086 (+ 910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.058088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.058090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.058161 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.058496 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.058498 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.058569 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.058676 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.058799 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.254@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
I20260620 20:18:35.069380 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.061706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.2.235.253:39357)
0620 20:18:35.061794 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.061797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.061991 (+ 194us) server_negotiation.cc:407] Connection header received
0620 20:18:35.062170 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.062173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.062217 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.062306 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.062825 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.063309 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.064169 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.064371 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.064468 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.066632 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.066635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.066638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.066659 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.067889 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.068304 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.068319 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.068321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.068388 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.068632 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.068635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.068636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.068770 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.068860 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.069129 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.069231 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:18:35.069409 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.061579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:35180)
0620 20:18:35.061889 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.061923 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.062052 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.062336 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.062342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.062631 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.062711 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.062716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.063406 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.063410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.063967 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.063975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.064473 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.064476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.064602 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.065074 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.065092 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.066504 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.067990 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.067999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.068003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.068202 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.068468 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.068471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.068473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.068549 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.068862 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.068869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.068966 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.069078 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.069236 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260620 20:18:35.070715 9096 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.235.253:39357: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
member_type: NON_VOTER
last_known_addr {
host: "127.2.235.253"
port: 39357
}
attrs {
promote: true
}
}
dest_uuid: "33ff3378d82e43a0894307e92d2a1997"
cas_config_opid_index: -1
I20260620 20:18:35.071200 9096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } }
I20260620 20:18:35.072518 9655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.073061 9602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:35.074077 9655 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.235.253:39357
I20260620 20:18:35.074226 9602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Refusing update from remote peer 33ff3378d82e43a0894307e92d2a1997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 20:18:35.074306 9655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.074385 9655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:35.074470 9560 master.cc:537] Master@127.2.235.253:39357 shutting down...
I20260620 20:18:35.074994 9655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:18:35.076620 9082 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 20:18:35.077443 9082 consensus_peers.cc:487] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 -> Peer 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253:39357): Couldn't send request to peer 1f79b50b6fdb46388c9e8d5cb38ba4fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:18:35.079555 9560 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:35.079811 9560 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:35.079921 9560 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: stopping tablet replica
W20260620 20:18:35.082111 9578 connection.cc:441] server connection from 127.0.0.1:35156 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:35156 (request call id 12) could send its response
I20260620 20:18:35.094117 9560 master.cc:552] Master@127.2.235.253:39357 shutdown complete.
I20260620 20:18:35.094165 9560 master_runner.cc:305] Clearing existing system tablet
I20260620 20:18:35.094406 9560 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.094468 9560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.095703 9560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 20:18:35.096519 9659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.096719 9560 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.096853 9560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:35.096891 9560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "130c9ea9f77dc5f2316918cad67573fd"
server_key_iv: "2bc1c8336550fc735b8a4716f5141496"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.099084 9560 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:35.100849 9560 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:18:35.100917 9560 log.cc:1196] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 20:18:35.101169 9560 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Deleting consensus metadata
I20260620 20:18:35.101368 9560 master_runner.cc:315] Copying system tablet from 127.2.235.254:34099
I20260620 20:18:35.102412 9560 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: tablet copy: Beginning tablet copy session from remote peer at address 127.2.235.254:34099
I20260620 20:18:35.110033 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.102987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.2.235.254:34099)
0620 20:18:35.103122 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.103127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.103272 (+ 145us) server_negotiation.cc:407] Connection header received
0620 20:18:35.103394 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.103396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.103431 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.103496 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.103974 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.104485 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.105608 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.105846 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.105921 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.107075 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.107079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.107081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.107103 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.108451 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.108917 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.108920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.108921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.108990 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.109244 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.109306 (+ 62us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.109308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.109470 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.109545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.109828 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.109923 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:18:35.110085 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.102919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:50634)
0620 20:18:35.103179 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.103193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.103313 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.103534 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.103537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.103741 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.103862 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.103868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.104577 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.104580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.105490 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.105500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.105949 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.105952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.106055 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.106536 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.106550 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.106946 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.108568 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.108572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.108574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.108819 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.109071 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.109074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.109076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.109162 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.109561 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.109564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.109661 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.109778 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.109918 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260620 20:18:35.110643 9116 tablet_copy_service.cc:140] P 33ff3378d82e43a0894307e92d2a1997: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1f79b50b6fdb46388c9e8d5cb38ba4fd ({username='oryx', principal='oryx/127.2.235.253@KRBTEST.COM'} at 127.0.0.1:50634)
I20260620 20:18:35.110829 9116 tablet_copy_service.cc:161] P 33ff3378d82e43a0894307e92d2a1997: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1f79b50b6fdb46388c9e8d5cb38ba4fd at {username='oryx', principal='oryx/127.2.235.253@KRBTEST.COM'} at 127.0.0.1:50634: session id = 1f79b50b6fdb46388c9e8d5cb38ba4fd-00000000000000000000000000000000
I20260620 20:18:35.112069 9116 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:18:35.113462 9560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:18:35.114979 9560 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: tablet copy: Starting download of 0 data blocks...
I20260620 20:18:35.115154 9560 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: tablet copy: Starting download of 1 WAL segments...
I20260620 20:18:35.116647 9560 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:18:35.118078 9116 tablet_copy_service.cc:342] P 33ff3378d82e43a0894307e92d2a1997: Request end of tablet copy session 1f79b50b6fdb46388c9e8d5cb38ba4fd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.235.253@KRBTEST.COM'} at 127.0.0.1:50634
I20260620 20:18:35.118158 9116 tablet_copy_service.cc:433] P 33ff3378d82e43a0894307e92d2a1997: ending tablet copy session 1f79b50b6fdb46388c9e8d5cb38ba4fd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1f79b50b6fdb46388c9e8d5cb38ba4fd
I20260620 20:18:35.124711 9560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:35.126187 9669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.126289 9667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.126247 9666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.126390 9560 server_base.cc:1034] running on GCE node
I20260620 20:18:35.126678 9560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:35.126978 9560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:35.128118 9560 hybrid_clock.cc:648] HybridClock initialized: now 1781986715128105 us; error 25 us; skew 500 ppm
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:35.129413 9560 init.cc:377] Logged in from keytab as oryx/127.2.235.253@KRBTEST.COM (short username oryx)
I20260620 20:18:35.129815 9560 webserver.cc:460] Webserver started at http://127.2.235.253:42967/ using document root <none> and password file <none>
I20260620 20:18:35.129987 9560 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.130038 9560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.131006 9560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 20:18:35.131809 9676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.132026 9560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:35.132081 9560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "130c9ea9f77dc5f2316918cad67573fd"
server_key_iv: "2bc1c8336550fc735b8a4716f5141496"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.132133 9560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.162025 9560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:35.163834 9560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:35.168742 9560 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.253:39357
I20260620 20:18:35.168762 9728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.253:39357 every 8 connection(s)
I20260620 20:18:35.169313 9560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
I20260620 20:18:35.170601 9729 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:18:35.171415 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Bootstrap starting.
I20260620 20:18:35.178857 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:18:35.179123 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: Bootstrap complete.
I20260620 20:18:35.179543 9729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } }
I20260620 20:18:35.179643 9729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f79b50b6fdb46388c9e8d5cb38ba4fd, State: Initialized, Role: LEARNER
I20260620 20:18:35.179745 9729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } }
I20260620 20:18:35.180004 9732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.180102 9732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: LEARNER
I20260620 20:18:35.180158 9729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:35.182626 9743 catalog_manager.cc:1259] Loaded cluster ID: 6a746e6d0f4341209871a0b8182d8522
I20260620 20:18:35.182677 9743 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: loading cluster ID for follower catalog manager: success
I20260620 20:18:35.184414 9743 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: acquiring CA information for follower catalog manager: success
I20260620 20:18:35.185508 9743 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:18:35.192268 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.183842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.2.235.253:39357)
0620 20:18:35.185704 (+ 1862us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.185707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.185718 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:35.185750 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.185752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.185809 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.185905 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.186556 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.187176 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.187935 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.188131 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.188223 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.189562 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.189570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.189572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.189591 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.190850 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.191223 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.191228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.191230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.191302 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.191549 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.191553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.191555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.191692 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.191820 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.192029 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.192129 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1806,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.2.235.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.252.keytab.
Entry for principal oryx/127.2.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.252.keytab.
I20260620 20:18:35.213513 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:40805
--webserver_interface=127.2.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.252.keytab
--principal=oryx/127.2.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:35.313361 9747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:35.313546 9747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:35.313599 9747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:35.317461 9747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:35.317533 9747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:35.317550 9747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:35.317592 9747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:35.317618 9747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:35.322290 9747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.252.keytab
--principal=oryx/127.2.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:35.323068 9747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:35.323945 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:35.331303 9753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.331331 9752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.331326 9755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.331872 9747 server_base.cc:1034] running on GCE node
I20260620 20:18:35.332360 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:35.333206 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:35.334405 9747 hybrid_clock.cc:648] HybridClock initialized: now 1781986715334393 us; error 23 us; skew 500 ppm
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:35.336869 9747 init.cc:377] Logged in from keytab as oryx/127.2.235.252@KRBTEST.COM (short username oryx)
I20260620 20:18:35.337724 9747 webserver.cc:460] Webserver started at http://127.2.235.252:38423/ using document root <none> and password file <none>
I20260620 20:18:35.338183 9747 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.338238 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.338444 9747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:35.340112 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/instance:
uuid: "09220ecc32d8489dbaf47934d329bb28"
format_stamp: "Formatted at 2026-06-20 20:18:35 on dist-test-slave-8z53"
server_key: "f379d2630de357a7d8cb0087bc596e0e"
server_key_iv: "6f8ff7a5dcf03a34f447500056b24106"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.340629 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal/instance:
uuid: "09220ecc32d8489dbaf47934d329bb28"
format_stamp: "Formatted at 2026-06-20 20:18:35 on dist-test-slave-8z53"
server_key: "f379d2630de357a7d8cb0087bc596e0e"
server_key_iv: "6f8ff7a5dcf03a34f447500056b24106"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.343703 9747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:18:35.345808 9762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.346887 9747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:35.347023 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
uuid: "09220ecc32d8489dbaf47934d329bb28"
format_stamp: "Formatted at 2026-06-20 20:18:35 on dist-test-slave-8z53"
server_key: "f379d2630de357a7d8cb0087bc596e0e"
server_key_iv: "6f8ff7a5dcf03a34f447500056b24106"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.347102 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.392956 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:35.395859 9747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:35.396047 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:35.402551 9747 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.252:40805
I20260620 20:18:35.402812 9814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.252:40805 every 8 connection(s)
I20260620 20:18:35.403396 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
I20260620 20:18:35.405936 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:35.408684 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9747
I20260620 20:18:35.409380 9815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.252@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.235.252@KRBTEST.COM
I20260620 20:18:35.421118 9287 heartbeater.cc:503] Master 127.2.235.254:34099 was elected leader, sending a full tablet report...
I20260620 20:18:35.421736 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.411163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.2.235.254:34099)
0620 20:18:35.411285 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.411289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.411452 (+ 163us) server_negotiation.cc:407] Connection header received
0620 20:18:35.411612 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.411618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.411672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.411743 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.412745 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.413589 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.414927 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.415137 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.415246 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.418026 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.418029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.418031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.418052 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.419660 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.420202 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.420208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.420211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.420295 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.420631 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.420635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.420636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.420828 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.420958 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.421489 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.421598 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:18:35.422640 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.410976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:50646)
0620 20:18:35.411323 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.411363 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.411478 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.411807 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.411812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.412243 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.412604 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.412615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.413701 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.413703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.414755 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.414766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.415274 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.415278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.415409 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.416035 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.416055 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.417863 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.419761 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.419768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.419790 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.420110 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.420373 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.420377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.420382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.420538 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.420944 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.420948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.421258 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.422072 (+ 814us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.422251 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":22,"spinlock_wait_cycles":1408,"thread_start_us":153,"threads_started":1}
I20260620 20:18:35.422645 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.410079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.2.235.252:40805)
0620 20:18:35.410428 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.410439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.410469 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:18:35.411021 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.411083 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.411445 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.411707 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.412549 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.413830 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.414860 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.415292 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.415395 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.418426 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.418446 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.418460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.418483 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.420127 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.420565 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.420569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.420570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.420662 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.420899 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.420905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.420909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.421234 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.421445 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.421645 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.421887 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":239,"spinlock_wait_cycles":2304,"thread_start_us":120,"threads_started":1}
I20260620 20:18:35.423614 9815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.252@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
I20260620 20:18:35.430516 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.424015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:35196)
0620 20:18:35.424139 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.424150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.424207 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.424410 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.424413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.424554 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.424624 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.424628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.425370 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.425373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.425906 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.425913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.426357 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.426381 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.426473 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.426920 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.426932 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.427911 (+ 979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.429204 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.429208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.429210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.429430 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.429681 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.429684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.429686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.429767 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.430081 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.430084 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.430157 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.430256 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.430391 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 20:18:35.430554 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.424114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.2.235.253:39357)
0620 20:18:35.424226 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.424234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.424244 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:18:35.424278 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.424281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.424314 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.424399 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.424747 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.425235 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.426016 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.426256 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.426362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.428027 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.428030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.428033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.428053 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.429107 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.429514 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.429519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.429520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.429590 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.429840 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.429844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.429846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.429985 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.430090 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.430311 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.430431 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:35.431134 9815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.252@KRBTEST.COM for oryx/127.2.235.252@KRBTEST.COM
I20260620 20:18:35.437429 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.431515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.0.0.1:53496)
0620 20:18:35.431611 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.431622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.431698 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.431847 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.431849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.431949 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.432000 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.432003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.432692 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.432694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.433148 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.433155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.433637 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.433640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.433736 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.434053 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.434062 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.435021 (+ 959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.436323 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.436326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.436327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.436511 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.436723 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.436725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.436726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.436792 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.437078 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.437080 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.437139 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.437209 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.437336 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":6}
I20260620 20:18:35.437552 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.431546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.2.235.252:40805)
0620 20:18:35.431677 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.431680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.431690 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:18:35.431760 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.431761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.431788 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.431842 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.432077 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.432595 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.433313 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.433533 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.433630 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.435120 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.435123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.435124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.435141 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.436226 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.436587 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.436589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.436590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.436651 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.436862 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.436864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.436865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.437007 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.437073 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.437307 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.437388 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":40}
I20260620 20:18:35.440131 9815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Bootstrap starting.
I20260620 20:18:35.442129 9815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:35.442755 9815 log.cc:824] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:35.444232 9815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: No bootstrap required, opened a new log
I20260620 20:18:35.446911 9815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } }
I20260620 20:18:35.447124 9815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:35.447186 9815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09220ecc32d8489dbaf47934d329bb28, State: Initialized, Role: FOLLOWER
I20260620 20:18:35.447568 9815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } }
I20260620 20:18:35.447944 9820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } } }
I20260620 20:18:35.448035 9820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:35.448516 9815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:18:35.452610 9831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:35.452688 9831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:35.454794 9820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:35.454950 9820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } }
I20260620 20:18:35.455657 9820 leader_election.cc:288] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099), 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253:39357)
I20260620 20:18:35.455751 9747 master_runner.cc:428] Detected that this master 09220ecc32d8489dbaf47934d329bb28 is joining an existing cluster
I20260620 20:18:35.455817 9747 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.235.252:40805
I20260620 20:18:35.456017 9106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09220ecc32d8489dbaf47934d329bb28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33ff3378d82e43a0894307e92d2a1997" is_pre_election: true
I20260620 20:18:35.456116 9703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09220ecc32d8489dbaf47934d329bb28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" is_pre_election: true
I20260620 20:18:35.456250 9703 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Handling vote request from an unknown peer 09220ecc32d8489dbaf47934d329bb28
I20260620 20:18:35.456390 9703 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 09220ecc32d8489dbaf47934d329bb28 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 20:18:35.456744 9765 leader_election.cc:302] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09220ecc32d8489dbaf47934d329bb28; no voters: 1f79b50b6fdb46388c9e8d5cb38ba4fd, 33ff3378d82e43a0894307e92d2a1997
I20260620 20:18:35.456987 9820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:35.457954 9820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:18:35.465202 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.458083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.0.0.1:53504)
0620 20:18:35.458590 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.458601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.458695 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.458970 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.458973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.459167 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.459250 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.459258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.460038 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.460040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.460838 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.460846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.461411 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.461414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.461488 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.461944 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.461960 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.462303 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.463837 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.463839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.463841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.464082 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.464333 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.464336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.464338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.464412 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.464741 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.464744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.464810 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.464919 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.465035 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":68,"threads_started":1}
I20260620 20:18:35.465230 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.457984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.2.235.252:40805)
0620 20:18:35.458165 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.458170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.458652 (+ 482us) server_negotiation.cc:407] Connection header received
0620 20:18:35.458769 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.458772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.458804 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.458842 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.459328 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.459948 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.460950 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.461226 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.461326 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.462425 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.462428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.462430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.462448 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.463730 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.464174 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.464179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.464182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.464248 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.464488 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.464493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.464494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.464658 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.464740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.464960 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.465114 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":119}
I20260620 20:18:35.475451 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9158
I20260620 20:18:35.481539 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.457955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:35202)
0620 20:18:35.458277 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.458288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.458418 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.458812 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.458814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.459069 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.459190 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.459196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.460080 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.460082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.460973 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.460984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.461639 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.461642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.461769 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.477866 (+ 16097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.477879 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.478313 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.480057 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.480060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.480062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.480264 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.480592 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.480594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.480596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.480665 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.481079 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.481082 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.481158 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.481259 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.481382 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":48,"thread_start_us":81,"threads_started":1}
I20260620 20:18:35.482219 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.457988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.2.235.253:39357)
0620 20:18:35.458089 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.458093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.458377 (+ 284us) server_negotiation.cc:407] Connection header received
0620 20:18:35.458487 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.458492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.458590 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.458688 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.459307 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.459989 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.461099 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.461436 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.461508 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.478447 (+ 16939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.478450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.478453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.478474 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.479948 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.480369 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.480372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.480374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.480478 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.480745 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.480747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.480749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.480935 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.481042 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.481920 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.482075 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:35.482527 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.458360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.2.235.254:34099)
0620 20:18:35.458465 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.458469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.459534 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 20:18:35.459674 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.459677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.459711 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.459749 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.460378 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.461063 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.462120 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.462439 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.462529 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.478362 (+ 15833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.478368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.478373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.478403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.480506 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.481037 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.481043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.481047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.481124 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.481420 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.481425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.481429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.481646 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.481735 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.482180 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.482306 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 20:18:35.482975 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:40531
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:34099
--webserver_port=33017
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:35.483232 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.458471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:50650)
0620 20:18:35.459462 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.459474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.459583 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.459857 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.459859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.460037 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.460267 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.460273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.461166 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.461169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.462006 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.462018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.462631 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.462634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.462716 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.477713 (+ 14997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.477731 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.478217 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.480659 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.480664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.480669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.480879 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.481220 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.481225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.481230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.481326 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.481827 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.481832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.481946 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.482841 (+ 895us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.483008 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":161,"thread_start_us":95,"threads_started":1}
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986714, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.254@KRBTEST.COM for oryx/127.2.235.252@KRBTEST.COM
I20260620 20:18:35.495607 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.487351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.0.0.1:53516)
0620 20:18:35.487450 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.487462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.487527 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.487877 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.487880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.488058 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.488141 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.488147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.489153 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.489156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.490003 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.490013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.490642 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.490646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.490736 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.491177 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.491190 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.492557 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.494279 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.494282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.494284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.494490 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.494728 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.494731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.494732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.494808 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.495162 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.495165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.495249 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.495351 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.495465 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 20:18:35.495626 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.487515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.2.235.252:40805)
0620 20:18:35.487625 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.487631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.487645 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:35.487710 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.487714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.487770 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.487854 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.488267 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.489035 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.490130 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.490529 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.490642 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.492694 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.492697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.492699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.492723 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.494169 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.494577 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.494580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.494582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.494648 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.494885 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.494893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.494894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.495045 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.495143 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.495416 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.495506 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:18:35.496781 9096 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.235.252:40805: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "09220ecc32d8489dbaf47934d329bb28"
member_type: NON_VOTER
last_known_addr {
host: "127.2.235.252"
port: 40805
}
attrs {
promote: true
}
}
dest_uuid: "33ff3378d82e43a0894307e92d2a1997"
cas_config_opid_index: 5
I20260620 20:18:35.497433 9096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } }
I20260620 20:18:35.498339 9654 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.499591 9654 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.235.252:40805
I20260620 20:18:35.499717 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.499788 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:35.500706 9789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Refusing update from remote peer 33ff3378d82e43a0894307e92d2a1997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:18:35.501192 9655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:35.502765 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.498440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:35210)
0620 20:18:35.498845 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.498857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.498940 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.499325 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.499330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.499348 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:35.499414 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.499419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.500295 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.500298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.501428 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.501438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.502390 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.502393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.502477 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.502516 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.502587 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.502662 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20260620 20:18:35.503135 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.498624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.2.235.253:39357)
0620 20:18:35.498774 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.498778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.499021 (+ 243us) server_negotiation.cc:407] Connection header received
0620 20:18:35.499065 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.499070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.499124 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.499208 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:35.499521 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.500190 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.501551 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.502122 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.502204 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.502337 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.502963 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.503031 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 20:18:35.503726 9703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Refusing update from remote peer 33ff3378d82e43a0894307e92d2a1997: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:18:35.504141 9655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 20:18:35.504830 9789 raft_consensus.cc:858] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } }
W20260620 20:18:35.504902 9789 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } }
I20260620 20:18:35.506084 9747 master.cc:537] Master@127.2.235.252:40805 shutting down...
I20260620 20:18:35.505827 9703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.508033 9732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ff3378d82e43a0894307e92d2a1997. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.508138 9732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: LEARNER
I20260620 20:18:35.508222 9732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.508289 9732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: LEARNER
I20260620 20:18:35.509194 9789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.514032 9820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ff3378d82e43a0894307e92d2a1997. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.514113 9820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 20:18:35.514179 9820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.514226 9820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 20:18:35.514230 9655 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: attempting to promote NON_VOTER 1f79b50b6fdb46388c9e8d5cb38ba4fd to VOTER
I20260620 20:18:35.514281 9820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: NON_VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: true } } }
I20260620 20:18:35.514325 9820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 20:18:35.514524 9655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } }
I20260620 20:18:35.515580 9703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 LEARNER]: Refusing update from remote peer 33ff3378d82e43a0894307e92d2a1997: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 20:18:35.516001 9747 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 20:18:35.516230 9747 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 20:18:35.516350 9747 tablet_replica.cc:330] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: stopping tablet replica
I20260620 20:18:35.516784 9655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 20:18:35.517967 9081 consensus_peers.cc:487] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 -> Peer 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252:40805): Couldn't send request to peer 09220ecc32d8489dbaf47934d329bb28. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:18:35.518720 9654 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.520036 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.520124 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:35.520810 9703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.522131 9843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.522223 9843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:35.522584 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.522684 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:35.537468 9747 master.cc:552] Master@127.2.235.252:40805 shutdown complete.
I20260620 20:18:35.537528 9747 master_runner.cc:305] Clearing existing system tablet
I20260620 20:18:35.537755 9747 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.537807 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.539083 9747 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 20:18:35.539935 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.540171 9747 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.540290 9747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:35.540321 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
uuid: "09220ecc32d8489dbaf47934d329bb28"
format_stamp: "Formatted at 2026-06-20 20:18:35 on dist-test-slave-8z53"
server_key: "f379d2630de357a7d8cb0087bc596e0e"
server_key_iv: "6f8ff7a5dcf03a34f447500056b24106"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.542112 9747 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:35.544420 9747 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:18:35.544505 9747 log.cc:1196] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 20:18:35.544866 9747 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Deleting consensus metadata
I20260620 20:18:35.545035 9747 master_runner.cc:315] Copying system tablet from 127.2.235.254:34099
I20260620 20:18:35.546141 9747 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: tablet copy: Beginning tablet copy session from remote peer at address 127.2.235.254:34099
I20260620 20:18:35.555027 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.546768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.2.235.254:34099)
0620 20:18:35.546900 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.546904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.547068 (+ 164us) server_negotiation.cc:407] Connection header received
0620 20:18:35.547177 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.547180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.547229 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.547321 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.547770 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.548590 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.549817 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.550163 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.550262 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.551490 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.551494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.551496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.551521 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.553317 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.553765 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.553769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.553770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.553850 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.554140 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.554142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.554144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.554346 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.554416 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.554736 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.554846 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:18:35.555752 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.546667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:50660)
0620 20:18:35.546967 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.546981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.547095 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.547335 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.547339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.547541 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.547648 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.547655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.548700 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.548704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.549682 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.549693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.550265 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.550268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.550360 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.550907 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.550920 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.551351 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.553430 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.553433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.553435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.553664 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.553955 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.553959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.553962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.554051 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.554486 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.554488 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.554574 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.555443 (+ 869us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.555582 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260620 20:18:35.556391 9116 tablet_copy_service.cc:140] P 33ff3378d82e43a0894307e92d2a1997: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 09220ecc32d8489dbaf47934d329bb28 ({username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:50660)
I20260620 20:18:35.556511 9116 tablet_copy_service.cc:161] P 33ff3378d82e43a0894307e92d2a1997: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 09220ecc32d8489dbaf47934d329bb28 at {username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:50660: session id = 09220ecc32d8489dbaf47934d329bb28-00000000000000000000000000000000
I20260620 20:18:35.557654 9116 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:18:35.558848 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:18:35.561051 9747 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: tablet copy: Starting download of 0 data blocks...
I20260620 20:18:35.561224 9747 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: tablet copy: Starting download of 1 WAL segments...
I20260620 20:18:35.562713 9747 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:18:35.564667 9116 tablet_copy_service.cc:342] P 33ff3378d82e43a0894307e92d2a1997: Request end of tablet copy session 09220ecc32d8489dbaf47934d329bb28-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:50660
I20260620 20:18:35.564747 9116 tablet_copy_service.cc:433] P 33ff3378d82e43a0894307e92d2a1997: ending tablet copy session 09220ecc32d8489dbaf47934d329bb28-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 09220ecc32d8489dbaf47934d329bb28
I20260620 20:18:35.571774 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:35.573351 9860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.573422 9863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.574241 9747 server_base.cc:1034] running on GCE node
W20260620 20:18:35.577704 9861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.577991 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:35.578299 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:35.579454 9747 hybrid_clock.cc:648] HybridClock initialized: now 1781986715579441 us; error 33 us; skew 500 ppm
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:35.581341 9747 init.cc:377] Logged in from keytab as oryx/127.2.235.252@KRBTEST.COM (short username oryx)
I20260620 20:18:35.581771 9747 webserver.cc:460] Webserver started at http://127.2.235.252:43621/ using document root <none> and password file <none>
I20260620 20:18:35.581947 9747 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.581984 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.583341 9747 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:35.584340 9870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.584587 9747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:35.584661 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
uuid: "09220ecc32d8489dbaf47934d329bb28"
format_stamp: "Formatted at 2026-06-20 20:18:35 on dist-test-slave-8z53"
server_key: "f379d2630de357a7d8cb0087bc596e0e"
server_key_iv: "6f8ff7a5dcf03a34f447500056b24106"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.584733 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.602759 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:35.605022 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:35.609946 9747 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.252:40805
I20260620 20:18:35.609979 9922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.252:40805 every 8 connection(s)
I20260620 20:18:35.610625 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
I20260620 20:18:35.612360 9923 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:18:35.613487 9923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Bootstrap starting.
I20260620 20:18:35.617637 9421 heartbeater.cc:503] Master 127.2.235.254:34099 was elected leader, sending a full tablet report...
W20260620 20:18:35.618376 9840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:35.618713 9840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:35.618798 9840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:18:35.621129 9923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:18:35.621424 9923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Bootstrap complete.
I20260620 20:18:35.621907 9923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } }
I20260620 20:18:35.622030 9923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09220ecc32d8489dbaf47934d329bb28, State: Initialized, Role: LEARNER
I20260620 20:18:35.622154 9923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } }
I20260620 20:18:35.622440 9929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.622560 9923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:35.622623 9929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: LEARNER
W20260620 20:18:35.624754 9840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:35.624873 9840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:35.625010 9840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:35.625119 9940 catalog_manager.cc:1259] Loaded cluster ID: 6a746e6d0f4341209871a0b8182d8522
I20260620 20:18:35.625228 9940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: loading cluster ID for follower catalog manager: success
I20260620 20:18:35.627466 9940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: acquiring CA information for follower catalog manager: success
I20260620 20:18:35.628821 9940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:18:35.631269 9840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:40531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=33017
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:35.632285 9840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:35.633292 9840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:35.642330 9942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.642339 9943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.642496 9840 server_base.cc:1034] running on GCE node
W20260620 20:18:35.642344 9945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.643175 9840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:35.643728 9840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:35.644912 9840 hybrid_clock.cc:648] HybridClock initialized: now 1781986715644899 us; error 24 us; skew 500 ppm
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:35.647612 9840 init.cc:377] Logged in from keytab as oryx/127.2.235.193@KRBTEST.COM (short username oryx)
I20260620 20:18:35.648303 9840 webserver.cc:460] Webserver started at http://127.2.235.193:33017/ using document root <none> and password file <none>
I20260620 20:18:35.648780 9840 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.648842 9840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.652251 9840 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:35.654176 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.655300 9840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:35.655386 9840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "df947cfb30144ad88d70d12a298746f8"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "3465588d0e29478e3bef09dcd4d6760b"
server_key_iv: "17798b79a6565d70c0cb6a644d77c7c8"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.655844 9840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.672993 9840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:35.675961 9840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:35.676093 9840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:35.676554 9840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:35.677340 9840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:35.677390 9840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.677426 9840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:35.677446 9840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.686134 9840 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40531
I20260620 20:18:35.686156 10065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40531 every 8 connection(s)
I20260620 20:18:35.687047 9840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:35.690083 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 9840
I20260620 20:18:35.690253 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9292
I20260620 20:18:35.696762 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:39229
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:34099
--webserver_port=34687
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.252@KRBTEST.COM
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:35.705920 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.689142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:60063 (local address 127.2.235.253:39357)
0620 20:18:35.689265 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.689270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.690098 (+ 828us) server_negotiation.cc:407] Connection header received
0620 20:18:35.692243 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.692247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.692300 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.692354 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.694354 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.695081 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697087 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.697410 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697476 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.701097 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.701100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.701103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.701126 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.703495 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704094 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.704097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.704099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.704194 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704611 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.704615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.704617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.704858 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.704922 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.705627 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.705747 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:35.705925 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.689290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:59927 (local address 127.2.235.252:40805)
0620 20:18:35.689705 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.689709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.689723 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:35.692243 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.692247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.692304 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.692357 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.694353 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.695312 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697087 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.697412 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697476 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.701506 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.701514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.701519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.701546 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.703231 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704094 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.704097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.704099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.704194 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704611 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.704615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.704617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.704843 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.704922 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.705627 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.705741 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":106,"threads_started":1}
I20260620 20:18:35.706017 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.689160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:53129 (local address 127.2.235.254:34099)
0620 20:18:35.689557 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.689561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.690148 (+ 587us) server_negotiation.cc:407] Connection header received
0620 20:18:35.692244 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.692250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.692309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.692361 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.694348 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.695156 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697087 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.697455 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.701760 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.701766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.701772 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.701804 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.703443 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704175 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.704181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.704185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.704279 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704635 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.704639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.704642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.704898 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.704991 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.705655 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.705802 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260620 20:18:35.707571 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.688982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:53129)
0620 20:18:35.690069 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.690083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.692037 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.692711 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.692720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.693469 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.694197 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.694213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.695288 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.695293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.696960 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.696973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.697716 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.698021 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.698971 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.698988 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.701600 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.703578 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.703583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.703587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.704051 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.704415 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.704423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.704516 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.705057 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.705062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.705345 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.706352 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.706548 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":1141,"spinlock_wait_cycles":24832,"thread_start_us":94,"threads_started":1}
I20260620 20:18:35.707799 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.688982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.2.235.193:60063)
0620 20:18:35.690007 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.690020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.692020 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.692710 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.692718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.693470 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.694195 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.694211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.695227 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.695230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.696955 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.696966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.697664 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.697996 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.698840 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.698850 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.700951 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.703602 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.703604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.703606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.703981 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.704307 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.704312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.704504 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.704984 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.704989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.705344 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.706341 (+ 997us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.706619 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":754,"spinlock_wait_cycles":47488,"thread_start_us":138,"threads_started":1}
I20260620 20:18:35.707568 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.688982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.2.235.193:59927)
0620 20:18:35.689540 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.689589 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.692020 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.692710 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.692718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.693470 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.694199 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.694211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.695430 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.695433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.696965 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.696971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.697664 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.697668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.698003 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.698610 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.698623 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.701364 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.703367 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.703372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.703386 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.703981 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.704307 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.704310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.704312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.704504 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.704983 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.704988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.705344 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.706124 (+ 780us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.706498 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":720,"spinlock_wait_cycles":30592,"thread_start_us":117,"threads_started":1}
I20260620 20:18:35.708976 10068 heartbeater.cc:347] Connected to a master server at 127.2.235.253:39357
I20260620 20:18:35.709005 10067 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34099
I20260620 20:18:35.709203 10067 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:35.708985 10066 heartbeater.cc:347] Connected to a master server at 127.2.235.252:40805
I20260620 20:18:35.709329 10066 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:35.709203 10068 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:35.709830 10067 heartbeater.cc:511] Master 127.2.235.254:34099 requested a full tablet report, sending...
I20260620 20:18:35.709832 10066 heartbeater.cc:511] Master 127.2.235.252:40805 requested a full tablet report, sending...
I20260620 20:18:35.709981 10068 heartbeater.cc:511] Master 127.2.235.253:39357 requested a full tablet report, sending...
I20260620 20:18:35.711084 9096 ts_manager.cc:194] Re-registered known tserver with Master: df947cfb30144ad88d70d12a298746f8 (127.2.235.193:40531)
I20260620 20:18:35.711265 9887 ts_manager.cc:194] Registered new tserver with Master: df947cfb30144ad88d70d12a298746f8 (127.2.235.193:40531)
I20260620 20:18:35.711264 9692 ts_manager.cc:194] Registered new tserver with Master: df947cfb30144ad88d70d12a298746f8 (127.2.235.193:40531)
I20260620 20:18:35.711791 9096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:53129
W20260620 20:18:35.804952 10074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:35.805172 10074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:35.805217 10074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:35.809461 10074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:35.809513 10074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:35.809571 10074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:35.814869 10074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:39229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=34687
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:35.815308 9555 heartbeater.cc:503] Master 127.2.235.254:34099 was elected leader, sending a full tablet report...
I20260620 20:18:35.815943 10074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:35.817137 10074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:35.826576 10079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.826660 10074 server_base.cc:1034] running on GCE node
W20260620 20:18:35.826551 10082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:35.826584 10080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:35.827196 10074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:35.827817 10074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:35.828994 10074 hybrid_clock.cc:648] HybridClock initialized: now 1781986715828972 us; error 44 us; skew 500 ppm
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:35.831487 10074 init.cc:377] Logged in from keytab as oryx/127.2.235.194@KRBTEST.COM (short username oryx)
I20260620 20:18:35.832309 10074 webserver.cc:460] Webserver started at http://127.2.235.194:34687/ using document root <none> and password file <none>
I20260620 20:18:35.832875 10074 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:35.832944 10074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:35.836182 10074 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:35.838126 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.839355 10074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:35.839457 10074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "a7599a977b004257a29b1f305af05e8f"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "2a722efe77f17d6e2de95e1c84eeada8"
server_key_iv: "994325ec95eeebc4caa76772010016eb"
server_key_version: "encryptionkey@0"
I20260620 20:18:35.839920 10074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:35.860109 10074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:35.862658 10074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:35.862844 10074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:35.863399 10074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:35.864238 10074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:35.864291 10074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.864346 10074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:35.864377 10074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:35.873014 10074 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:39229
I20260620 20:18:35.873046 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:39229 every 8 connection(s)
I20260620 20:18:35.873827 10074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:35.882740 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 10074
I20260620 20:18:35.882913 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9426
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.252@KRBTEST.COM
Jun 20 20:18:35 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986715, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
I20260620 20:18:35.889416 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:40047
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:34099
--webserver_port=41551
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:35.891849 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.875634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:58335 (local address 127.2.235.252:40805)
0620 20:18:35.875732 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.875736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.877115 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 20:18:35.877679 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.877682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.877740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.877812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.878985 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.879800 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881094 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.881465 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881554 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.887749 (+ 6195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.887753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.887755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.887779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.889882 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890364 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.890370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.890373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.890455 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890780 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.890783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.890785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.890984 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.891046 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.891562 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.891681 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:18:35.891944 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.875923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:38711 (local address 127.2.235.253:39357)
0620 20:18:35.876073 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.876077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.876846 (+ 769us) server_negotiation.cc:407] Connection header received
0620 20:18:35.877689 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.877693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.877742 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.877793 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.879007 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.879512 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881126 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.881438 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881549 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.887781 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.887787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.887792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.887832 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.890339 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890780 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.890783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.890785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.890861 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.891132 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.891135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.891137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.891314 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.891381 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.891683 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.891790 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:18:35.892550 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.875356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:40861 (local address 127.2.235.254:34099)
0620 20:18:35.875471 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.875475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.876449 (+ 974us) server_negotiation.cc:407] Connection header received
0620 20:18:35.877689 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.877693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.877743 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.877793 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:35.879018 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.879800 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.880924 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.881319 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881416 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.887749 (+ 6333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:35.887753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:35.887755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:35.887780 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:35.889582 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890180 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.890186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.890191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.890273 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890687 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:35.890692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:35.890695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:35.890925 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:35.890999 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.892323 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.892428 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:18:35.892720 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.875769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.194:40861)
0620 20:18:35.876316 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.876363 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.877476 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.877983 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.877988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.878407 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.878864 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.878871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.879952 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.879955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.880798 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.880807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.881668 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.881822 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.882488 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.882507 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.887602 (+ 5095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.889705 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.889712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.889728 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.890070 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.890395 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.890405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.890582 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.891076 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.891081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.891347 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.891516 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.891820 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":268,"spinlock_wait_cycles":23680,"thread_start_us":126,"threads_started":1}
I20260620 20:18:35.892724 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.875769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.2.235.194:38711)
0620 20:18:35.876740 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.876752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.877476 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.877983 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.877990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.878407 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.878862 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.878871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.879686 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.879688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.881005 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.881012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.881579 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.881851 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.883709 (+ 1858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.883720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.887639 (+ 3919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.890455 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.890460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.890680 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.890964 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.890969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.891042 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.891446 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.891451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.891527 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.892110 (+ 583us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.892226 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":379,"spinlock_wait_cycles":5504,"thread_start_us":89,"threads_started":1}
I20260620 20:18:35.892968 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.875769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.2.235.194:58335)
0620 20:18:35.877001 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.877014 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.877481 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.877981 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.877987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.878401 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:35.878848 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.878860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.879916 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.879918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.880981 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.880991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.881680 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.881683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.881858 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.882403 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:35.882417 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:35.887578 (+ 5161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:35.890003 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.890008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.890242 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.890559 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:35.890562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:35.890563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:35.890689 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:35.891109 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:35.891114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:35.891345 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.891500 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.891874 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":497,"spinlock_wait_cycles":12288,"thread_start_us":99,"threads_started":1}
I20260620 20:18:35.893853 10205 heartbeater.cc:347] Connected to a master server at 127.2.235.253:39357
I20260620 20:18:35.893879 10204 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34099
I20260620 20:18:35.894050 10205 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:35.894052 10204 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:35.894577 10205 heartbeater.cc:511] Master 127.2.235.253:39357 requested a full tablet report, sending...
I20260620 20:18:35.894577 10204 heartbeater.cc:511] Master 127.2.235.254:34099 requested a full tablet report, sending...
I20260620 20:18:35.894721 10203 heartbeater.cc:347] Connected to a master server at 127.2.235.252:40805
I20260620 20:18:35.894788 10203 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:35.894928 10203 heartbeater.cc:511] Master 127.2.235.252:40805 requested a full tablet report, sending...
I20260620 20:18:35.895565 9096 ts_manager.cc:194] Re-registered known tserver with Master: a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:35.895596 9887 ts_manager.cc:194] Registered new tserver with Master: a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:35.895623 9692 ts_manager.cc:194] Registered new tserver with Master: a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:35.896312 9096 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:40861
I20260620 20:18:35.975670 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.972719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.0.0.1:53526)
0620 20:18:35.972856 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:35.972872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:35.972987 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:35.973263 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:35.973267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:35.973285 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:35.973380 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.973385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.974029 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.974034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.974674 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:35.974681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.975282 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.975285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:35.975383 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.975404 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:18:35.975473 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:18:35.975547 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 20:18:35.975670 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:35.972796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.2.235.252:40805)
0620 20:18:35.972915 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:35.972920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:35.972934 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:35.973092 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:35.973097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:35.973147 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:35.973231 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:35.973488 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.973929 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.974791 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:35.975169 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:35.975271 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:35.975429 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:35.975493 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:18:35.975557 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:18:35.976269 9897 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 20:18:35.976548 9929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33ff3378d82e43a0894307e92d2a1997. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:35.976715 9929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: LEARNER
W20260620 20:18:35.993403 10210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:35.993574 10210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:35.993607 10210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:35.997077 10210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:35.997113 10210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:35.997159 10210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:36.001314 10210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:40047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=41551
--tserver_master_addrs=127.2.235.254:34099,127.2.235.253:39357,127.2.235.252:40805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:36.001981 10210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:36.002763 10210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:36.010861 10215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:36.010870 10218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:36.010876 10216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:36.011067 10210 server_base.cc:1034] running on GCE node
I20260620 20:18:36.011380 10210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:36.011933 10210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:36.013113 10210 hybrid_clock.cc:648] HybridClock initialized: now 1781986716013092 us; error 40 us; skew 500 ppm
Jun 20 20:18:36 dist-test-slave-8z53 krb5kdc[9051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986716, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:36.015357 10210 init.cc:377] Logged in from keytab as oryx/127.2.235.195@KRBTEST.COM (short username oryx)
I20260620 20:18:36.016093 10210 webserver.cc:460] Webserver started at http://127.2.235.195:41551/ using document root <none> and password file <none>
I20260620 20:18:36.016606 10210 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:36.016707 10210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:36.020110 10210 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:36.021799 10225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:36.023095 10210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:36.023218 10210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "8b39446a91e14c1e8c0c01e650194f95"
format_stamp: "Formatted at 2026-06-20 20:18:34 on dist-test-slave-8z53"
server_key: "d0335a29ef64a4eedb5169b1b85b5846"
server_key_iv: "8554f2ba292e004a4d1363996341b345"
server_key_version: "encryptionkey@0"
I20260620 20:18:36.023893 10210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:36.039384 10210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:36.041671 10210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:36.041790 10210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:36.042210 10210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:36.042904 10210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:36.042944 10210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:36.043002 10210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:36.043030 10210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:36.051265 10210 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:40047
I20260620 20:18:36.051292 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:40047 every 8 connection(s)
I20260620 20:18:36.052090 10210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:36.058907 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 10210
Jun 20 20:18:36 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986716, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
Jun 20 20:18:36 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986716, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.253@KRBTEST.COM
Jun 20 20:18:36 dist-test-slave-8z53 krb5kdc[9051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986716, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.252@KRBTEST.COM
I20260620 20:18:36.069703 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.053594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:42175 (local address 127.2.235.254:34099)
0620 20:18:36.053701 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.053705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.055201 (+ 1496us) server_negotiation.cc:407] Connection header received
0620 20:18:36.055656 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.055660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.055709 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.055759 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.057048 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.057911 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059403 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.059734 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059797 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.063995 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.063999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.064001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.064024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.065666 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.067895 (+ 2229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.067899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.067901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.067980 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.068514 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.068520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.068522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.068729 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.068800 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.069455 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.069556 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:36.070838 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.053866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:45949 (local address 127.2.235.253:39357)
0620 20:18:36.054079 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.054083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.055048 (+ 965us) server_negotiation.cc:407] Connection header received
0620 20:18:36.055641 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.055644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.055701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.055804 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.057035 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.057591 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059402 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.059724 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059797 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.066296 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.066300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.066303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.066339 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.068680 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.069567 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.069571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.069574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.069655 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.069962 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.069968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.069970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.070159 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.070223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.070558 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.070662 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 20:18:36.071336 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.054036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.2.235.195:45949)
0620 20:18:36.054958 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.054968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.055450 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.055993 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.055999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.056354 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:36.056873 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.056885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.057756 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.057759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.059268 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.059278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.059975 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.060242 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.061207 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:36.061231 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:36.066069 (+ 4838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:36.069173 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:36.069178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:36.069181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:36.069453 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:36.069759 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:36.069761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:36.069763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:36.069857 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:36.070291 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:36.070297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:36.070384 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.070964 (+ 580us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.071102 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":381,"spinlock_wait_cycles":52096,"thread_start_us":203,"threads_started":1}
I20260620 20:18:36.071342 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.054036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:42175)
0620 20:18:36.055132 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.055140 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.055449 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.055992 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.056001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.056370 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:36.056891 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.056897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.058020 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.058023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.059281 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.059287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.059975 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.060335 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.061228 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:36.061242 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:36.063825 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:36.065775 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:36.065782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:36.065793 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:36.067810 (+ 2017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:36.068223 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:36.068225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:36.068228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:36.068395 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:36.068884 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:36.068891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:36.069161 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.069984 (+ 823us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.070265 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":571,"spinlock_wait_cycles":8832,"thread_start_us":181,"threads_started":1}
I20260620 20:18:36.072418 10341 heartbeater.cc:347] Connected to a master server at 127.2.235.253:39357
I20260620 20:18:36.072450 10340 heartbeater.cc:347] Connected to a master server at 127.2.235.254:34099
I20260620 20:18:36.072607 10341 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:36.073089 10341 heartbeater.cc:511] Master 127.2.235.253:39357 requested a full tablet report, sending...
I20260620 20:18:36.073124 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.054036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.2.235.195:60523)
0620 20:18:36.054533 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.054575 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.055449 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.055992 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.056001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.056368 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:36.056889 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.056897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.058307 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.058310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.059286 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.059297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.060012 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.060016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.060234 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.061200 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:36.061228 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:36.066400 (+ 5172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:36.071619 (+ 5219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:36.071624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:36.071627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:36.071849 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:36.072125 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:36.072128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:36.072130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:36.072201 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:36.072609 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:36.072613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:36.072706 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.072801 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.072955 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":266,"spinlock_wait_cycles":20096,"thread_start_us":119,"threads_started":1}
I20260620 20:18:36.073133 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.053833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:60523 (local address 127.2.235.252:40805)
0620 20:18:36.053998 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.054002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.054655 (+ 653us) server_negotiation.cc:407] Connection header received
0620 20:18:36.055656 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.055660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.055713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.055759 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.057048 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.057920 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059418 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.059745 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.059840 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.066527 (+ 6687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.066530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.066533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.066556 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.069051 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.071948 (+ 2897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.071951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.071953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.072026 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.072290 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.072293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.072295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.072474 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.072540 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.072875 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.072999 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:18:36.072607 10340 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:36.073761 10340 heartbeater.cc:511] Master 127.2.235.254:34099 requested a full tablet report, sending...
I20260620 20:18:36.073819 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.060805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.2.235.254:34099)
0620 20:18:36.062135 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.062139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.062152 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:36.062190 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.062194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.062240 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.062337 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.063135 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.063948 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.065215 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.065584 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.066074 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.067361 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.067366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.067370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.067397 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.070832 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.071707 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.071709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.071711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.071782 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.072648 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.072652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.072655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.072839 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.073426 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.073512 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.073639 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":64,"threads_started":1}
I20260620 20:18:36.073825 10339 heartbeater.cc:347] Connected to a master server at 127.2.235.252:40805
I20260620 20:18:36.073891 10339 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:36.074043 10339 heartbeater.cc:511] Master 127.2.235.252:40805 requested a full tablet report, sending...
I20260620 20:18:36.074368 9692 ts_manager.cc:194] Registered new tserver with Master: 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047)
I20260620 20:18:36.074420 9095 ts_manager.cc:194] Re-registered known tserver with Master: 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047)
I20260620 20:18:36.074689 9887 ts_manager.cc:194] Registered new tserver with Master: 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047)
I20260620 20:18:36.075444 9095 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:42175
I20260620 20:18:36.076401 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.060989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.2.235.252:40805)
0620 20:18:36.062885 (+ 1896us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.062889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.062900 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:36.062932 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.062935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.062972 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.063060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.063936 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.064724 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.065902 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.066254 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.066364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.068069 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.068074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.068082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.068117 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.070126 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.072144 (+ 2018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.072148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.072150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.072230 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.075337 (+ 3107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.075340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.075342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.075529 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.075658 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.076136 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.076242 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1821,"thread_start_us":72,"threads_started":1}
I20260620 20:18:36.079402 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.061594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.2.235.253:39357)
0620 20:18:36.061846 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.061850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.062665 (+ 815us) server_negotiation.cc:407] Connection header received
0620 20:18:36.062821 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.062825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.062877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.062958 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.063794 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.068125 (+ 4331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.070214 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.070494 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.070580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.075795 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.075801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.075804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.075854 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.077894 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.078353 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.078356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.078357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.078433 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.078683 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.078685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.078686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.078822 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.078911 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.079124 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.079209 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
I20260620 20:18:36.077425 9095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:36.081346 9095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:36.083645 9847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: NON_VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: true } } }
I20260620 20:18:36.083787 9847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:36.085206 9847 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: attempting to promote NON_VOTER 09220ecc32d8489dbaf47934d329bb28 to VOTER
I20260620 20:18:36.085561 9847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } }
I20260620 20:18:36.086633 9897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 LEARNER]: Refusing update from remote peer 33ff3378d82e43a0894307e92d2a1997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 20:18:36.086640 9703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Refusing update from remote peer 33ff3378d82e43a0894307e92d2a1997: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 20:18:36.086969 10358 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:18:36.087145 10357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:18:36.088572 10358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.089083 9703 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.089077 9897 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.090085 10358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.090173 10358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:36.090296 10358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.090365 10358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:36.090381 9929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.090484 9929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:36.090459 10358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.090520 10358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:36.090566 10366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33ff3378d82e43a0894307e92d2a1997" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:36.090668 10366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:36.100891 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.096381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40047 (local address 127.0.0.1:38024)
0620 20:18:36.096515 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.096531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.096610 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.097800 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.097803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.097843 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:36.097940 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.097947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.098970 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.098973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.099840 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.099848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.100549 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.100552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.100633 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.100652 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.100723 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.100785 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:18:36.100953 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.096469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.2.235.195:40047)
0620 20:18:36.097297 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.097318 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.097343 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:18:36.097498 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.097505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.097678 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.097843 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:36.098064 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.098860 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.099941 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.100424 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.100549 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.100671 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.100750 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.100835 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":98,"threads_started":1}
I20260620 20:18:36.102367 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.096927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:39229 (local address 127.0.0.1:41914)
0620 20:18:36.098054 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.098066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.098144 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.098586 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.098589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.098605 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:36.098679 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.098684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.099867 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.099870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.100982 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.100994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.102022 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.102025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.102105 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.102124 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.102165 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.102253 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260620 20:18:36.102968 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.097304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.2.235.194:39229)
0620 20:18:36.097675 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.097683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.098134 (+ 451us) server_negotiation.cc:407] Connection header received
0620 20:18:36.098246 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.098252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.098435 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.098587 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:36.098767 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.099525 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.101121 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.101909 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.102612 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.102724 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.102765 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.102847 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":91,"threads_started":1}
I20260620 20:18:36.103556 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.096849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40531 (local address 127.0.0.1:39018)
0620 20:18:36.097097 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.097109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.097216 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.099638 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.099640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.099674 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:36.099734 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.099737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.101101 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.101105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.102142 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.102152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.103219 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.103222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.103300 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.103319 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.103389 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.103449 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260620 20:18:36.103615 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.097246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.2.235.193:40531)
0620 20:18:36.098380 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.098391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.098413 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:18:36.098483 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.098490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.098660 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.098786 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:36.099832 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.100918 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.102269 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.103118 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.103208 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.103328 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.103413 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.103488 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":89,"threads_started":1}
I20260620 20:18:36.103590 10273 tablet_service.cc:1467] Processing CreateTablet for tablet 120e8df2b0fb4eb49615747b20f26f9a (DEFAULT_TABLE table=test-table [id=3249b6bc645d47cca4a86c149d731ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:36.104599 10273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120e8df2b0fb4eb49615747b20f26f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:36.105875 10000 tablet_service.cc:1467] Processing CreateTablet for tablet 120e8df2b0fb4eb49615747b20f26f9a (DEFAULT_TABLE table=test-table [id=3249b6bc645d47cca4a86c149d731ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:36.106775 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120e8df2b0fb4eb49615747b20f26f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:36.107143 10137 tablet_service.cc:1467] Processing CreateTablet for tablet 120e8df2b0fb4eb49615747b20f26f9a (DEFAULT_TABLE table=test-table [id=3249b6bc645d47cca4a86c149d731ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:36.107949 10137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120e8df2b0fb4eb49615747b20f26f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:36.111955 10372 tablet_bootstrap.cc:492] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Bootstrap starting.
I20260620 20:18:36.113111 10373 tablet_bootstrap.cc:492] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Bootstrap starting.
I20260620 20:18:36.114030 10372 tablet_bootstrap.cc:654] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:36.114121 10374 tablet_bootstrap.cc:492] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Bootstrap starting.
I20260620 20:18:36.114825 10372 log.cc:824] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:36.115188 10373 tablet_bootstrap.cc:654] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:36.115840 10374 tablet_bootstrap.cc:654] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:36.115974 10373 log.cc:824] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:36.116657 10374 log.cc:824] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:36.116963 10372 tablet_bootstrap.cc:492] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: No bootstrap required, opened a new log
I20260620 20:18:36.117200 10372 ts_tablet_manager.cc:1397] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:18:36.117946 10373 tablet_bootstrap.cc:492] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: No bootstrap required, opened a new log
I20260620 20:18:36.118116 10373 ts_tablet_manager.cc:1397] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:18:36.118212 10374 tablet_bootstrap.cc:492] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: No bootstrap required, opened a new log
I20260620 20:18:36.118403 10374 ts_tablet_manager.cc:1397] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:18:36.121337 10374 raft_consensus.cc:348] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.121577 10374 raft_consensus.cc:374] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:36.121632 10374 raft_consensus.cc:729] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7599a977b004257a29b1f305af05e8f, State: Initialized, Role: FOLLOWER
I20260620 20:18:36.121469 10372 raft_consensus.cc:348] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.121698 10372 raft_consensus.cc:374] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:36.121749 10372 raft_consensus.cc:729] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39446a91e14c1e8c0c01e650194f95, State: Initialized, Role: FOLLOWER
I20260620 20:18:36.122064 10374 consensus_queue.cc:260] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.122120 10372 consensus_queue.cc:260] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.122246 10373 raft_consensus.cc:348] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.122529 10373 raft_consensus.cc:374] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:36.122612 10373 raft_consensus.cc:729] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df947cfb30144ad88d70d12a298746f8, State: Initialized, Role: FOLLOWER
I20260620 20:18:36.122978 10374 ts_tablet_manager.cc:1428] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:18:36.122982 10372 ts_tablet_manager.cc:1428] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:18:36.123036 10373 consensus_queue.cc:260] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.123168 10204 heartbeater.cc:503] Master 127.2.235.254:34099 was elected leader, sending a full tablet report...
I20260620 20:18:36.123270 10340 heartbeater.cc:503] Master 127.2.235.254:34099 was elected leader, sending a full tablet report...
I20260620 20:18:36.123826 10373 ts_tablet_manager.cc:1428] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 20:18:36.125458 10206 tablet.cc:2369] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:36.125626 10067 heartbeater.cc:503] Master 127.2.235.254:34099 was elected leader, sending a full tablet report...
I20260620 20:18:36.170552 10380 raft_consensus.cc:482] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:36.170722 10380 raft_consensus.cc:504] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.171670 10380 leader_election.cc:288] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047), a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:36.175664 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.171846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40047 (local address 127.2.235.193:54931)
0620 20:18:36.172011 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.172026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.172106 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.172374 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.172378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.172392 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:36.172511 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.172517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.173542 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.173544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.174575 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.174585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.175226 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.175230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.175327 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.175349 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.175443 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.175510 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":5}
I20260620 20:18:36.175858 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.171903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:39229 (local address 127.2.235.193:42595)
0620 20:18:36.172120 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:36.172134 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:36.172205 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:36.172547 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:36.172550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:36.172565 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:36.172677 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.172682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.173397 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.173403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.174581 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:36.174586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.175465 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.175468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:36.175550 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.175569 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:18:36.175642 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:18:36.175714 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":43}
I20260620 20:18:36.175889 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.172210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:42595 (local address 127.2.235.194:39229)
0620 20:18:36.172338 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.172345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.172357 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:36.172399 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.172401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.172446 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.172537 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:36.172780 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.173291 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.174699 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.175344 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.175482 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.175531 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.175675 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.175749 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:36.176468 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.171962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:54931 (local address 127.2.235.195:40047)
0620 20:18:36.172069 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.172075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.172087 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:36.172227 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.172232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.172280 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.172354 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:36.172638 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.173444 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.174730 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.175105 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.175226 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.175275 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.176278 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.176361 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:18:36.176455 10157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120e8df2b0fb4eb49615747b20f26f9a" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7599a977b004257a29b1f305af05e8f" is_pre_election: true
I20260620 20:18:36.176720 10157 raft_consensus.cc:2454] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 0.
I20260620 20:18:36.177004 10293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120e8df2b0fb4eb49615747b20f26f9a" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39446a91e14c1e8c0c01e650194f95" is_pre_election: true
I20260620 20:18:36.177172 9954 leader_election.cc:302] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7599a977b004257a29b1f305af05e8f, df947cfb30144ad88d70d12a298746f8; no voters:
I20260620 20:18:36.177269 10293 raft_consensus.cc:2454] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 0.
I20260620 20:18:36.177431 10380 raft_consensus.cc:2781] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:36.177532 10380 raft_consensus.cc:482] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:36.177577 10380 raft_consensus.cc:3037] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:36.178527 10380 raft_consensus.cc:504] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.178871 10380 leader_election.cc:288] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047), a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:36.179160 10293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120e8df2b0fb4eb49615747b20f26f9a" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39446a91e14c1e8c0c01e650194f95"
I20260620 20:18:36.179162 10157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120e8df2b0fb4eb49615747b20f26f9a" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7599a977b004257a29b1f305af05e8f"
I20260620 20:18:36.179266 10293 raft_consensus.cc:3037] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:36.179266 10157 raft_consensus.cc:3037] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:36.180419 10293 raft_consensus.cc:2454] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 1.
I20260620 20:18:36.180419 10157 raft_consensus.cc:2454] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 1.
I20260620 20:18:36.180668 9954 leader_election.cc:302] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7599a977b004257a29b1f305af05e8f, df947cfb30144ad88d70d12a298746f8; no voters:
I20260620 20:18:36.180941 10380 raft_consensus.cc:2781] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:36.181097 10380 raft_consensus.cc:686] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 1 LEADER]: Becoming Leader. State: Replica: df947cfb30144ad88d70d12a298746f8, State: Running, Role: LEADER
I20260620 20:18:36.181404 10380 consensus_queue.cc:237] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } }
I20260620 20:18:36.184688 9096 catalog_manager.cc:5495] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df947cfb30144ad88d70d12a298746f8 (127.2.235.193). New cstate: current_term: 1 leader_uuid: "df947cfb30144ad88d70d12a298746f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } health_report { overall_health: UNKNOWN } } }
W20260620 20:18:36.188361 10069 tablet.cc:2369] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:36.196251 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.193365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.2.235.193:40531)
0620 20:18:36.193456 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.193460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.193473 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:36.193591 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.193593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.193630 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.193694 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:36.193910 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.194338 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.195124 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.195297 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.195368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.195628 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:36.195982 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:36.196065 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.196130 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.196171 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:18:36.201107 10157 raft_consensus.cc:1261] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Refusing update from remote peer df947cfb30144ad88d70d12a298746f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:36.201160 10293 raft_consensus.cc:1261] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Refusing update from remote peer df947cfb30144ad88d70d12a298746f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:36.201745 10381 consensus_queue.cc:1035] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:36.201937 10380 consensus_queue.cc:1035] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:36.214423 10385 mvcc.cc:205] Tried to move back new op lower bound from 7299017589553840128 to 7299017589481762816. Current Snapshot: MvccSnapshot[applied={T|T < 7299017589553840128 or (T in {7299017589553840128})}]
I20260620 20:18:36.218425 9095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:36.218539 9095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:36.221827 9095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33ff3378d82e43a0894307e92d2a1997: Sending DeleteTablet for 3 replicas of tablet 120e8df2b0fb4eb49615747b20f26f9a
I20260620 20:18:36.222388 10273 tablet_service.cc:1514] Processing DeleteTablet for tablet 120e8df2b0fb4eb49615747b20f26f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:36 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:38024
I20260620 20:18:36.222467 10137 tablet_service.cc:1514] Processing DeleteTablet for tablet 120e8df2b0fb4eb49615747b20f26f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:36 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:41914
I20260620 20:18:36.222759 10394 tablet_replica.cc:330] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: stopping tablet replica
I20260620 20:18:36.222774 10000 tablet_service.cc:1514] Processing DeleteTablet for tablet 120e8df2b0fb4eb49615747b20f26f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:36 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:39018
I20260620 20:18:36.222910 10395 tablet_replica.cc:330] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: stopping tablet replica
I20260620 20:18:36.223105 10394 raft_consensus.cc:2229] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:36.223163 10395 raft_consensus.cc:2229] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:36.223366 10394 raft_consensus.cc:2258] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:36.223403 10395 raft_consensus.cc:2258] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:36.225071 10396 tablet_replica.cc:330] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: stopping tablet replica
I20260620 20:18:36.225324 10396 raft_consensus.cc:2229] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:36.225811 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9064
I20260620 20:18:36.226876 10394 ts_tablet_manager.cc:1905] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:36.230093 10394 ts_tablet_manager.cc:1918] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:36.230180 10394 log.cc:1196] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/120e8df2b0fb4eb49615747b20f26f9a
I20260620 20:18:36.230477 10394 ts_tablet_manager.cc:1939] T 120e8df2b0fb4eb49615747b20f26f9a P 8b39446a91e14c1e8c0c01e650194f95: Deleting consensus metadata
I20260620 20:18:36.234005 10395 ts_tablet_manager.cc:1905] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:36.236274 10395 ts_tablet_manager.cc:1918] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:36.236356 10395 log.cc:1196] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/120e8df2b0fb4eb49615747b20f26f9a
I20260620 20:18:36.236646 10395 ts_tablet_manager.cc:1939] T 120e8df2b0fb4eb49615747b20f26f9a P a7599a977b004257a29b1f305af05e8f: Deleting consensus metadata
I20260620 20:18:36.236814 10396 raft_consensus.cc:2258] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 20:18:36.237538 10092 connection.cc:441] server connection from 127.0.0.1:41914 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41914 (request call id 1) could send its response
I20260620 20:18:36.238175 10396 ts_tablet_manager.cc:1905] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:36.241223 10396 ts_tablet_manager.cc:1918] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:36.241341 10396 log.cc:1196] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/120e8df2b0fb4eb49615747b20f26f9a
I20260620 20:18:36.241690 10396 ts_tablet_manager.cc:1939] T 120e8df2b0fb4eb49615747b20f26f9a P df947cfb30144ad88d70d12a298746f8: Deleting consensus metadata
W20260620 20:18:36.242671 9956 connection.cc:441] server connection from 127.0.0.1:39018 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39018 (request call id 1) could send its response
I20260620 20:18:36.243248 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.225350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.2.235.252:40805)
0620 20:18:36.225465 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.225469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.231634 (+ 6165us) server_negotiation.cc:407] Connection header received
0620 20:18:36.231812 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.231817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.231871 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.231920 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.232990 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.233801 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.235452 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.235847 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.235925 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.239282 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.239286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.239288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.239312 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.241094 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.241637 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.241643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.241647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.241732 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.242073 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.242076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.242077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.242268 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.242388 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.242998 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.243103 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:18:36.250428 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.237321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.2.235.252:40805)
0620 20:18:36.237436 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.237440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.241761 (+ 4321us) server_negotiation.cc:407] Connection header received
0620 20:18:36.241976 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.241979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.242017 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.242062 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.242913 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.243601 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.245089 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.245450 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.245517 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.247223 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.247227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.247229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.247268 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.248759 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.249189 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.249194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.249196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.249383 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.249665 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.249669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.249670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.249866 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.249966 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.250182 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.250277 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:18:36.250913 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:36.237535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.2.235.253:39357)
0620 20:18:36.237755 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:36.237759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:36.243839 (+ 6080us) server_negotiation.cc:407] Connection header received
0620 20:18:36.243954 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:36.243957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:36.244025 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:36.244113 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:36.244707 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.245216 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.246052 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:36.246408 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:36.246507 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:36.247816 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:36.247819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:36.247822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:36.247850 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:36.249369 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.249752 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.249755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.249757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.249820 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:36.250096 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:36.250099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:36.250101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:36.250280 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:36.250364 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:36.250659 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 20:18:36.250766 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 20:18:37.211608 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.211195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:57929)
0620 20:18:37.211446 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.211519 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
W20260620 20:18:37.211934 10067 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:34099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.213500 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.213167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:47867)
0620 20:18:37.213368 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.213436 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
W20260620 20:18:37.213792 10340 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:34099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.214462 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.214082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.194:33839)
0620 20:18:37.214284 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.214354 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260620 20:18:37.214915 10204 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:34099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.722373 10408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 33ff3378d82e43a0894307e92d2a1997)
I20260620 20:18:37.722506 10408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } }
I20260620 20:18:37.723027 10408 leader_election.cc:288] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099), 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253:39357)
I20260620 20:18:37.723537 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.723129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57690)
0620 20:18:37.723359 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.723461 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
W20260620 20:18:37.723749 9874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.724601 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.724380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57694)
0620 20:18:37.724482 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.724536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:18:37.724718 9874 leader_election.cc:334] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099): Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.725389 10413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 33ff3378d82e43a0894307e92d2a1997)
I20260620 20:18:37.725575 10413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } }
I20260620 20:18:37.726555 10413 leader_election.cc:288] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099), 09220ecc32d8489dbaf47934d329bb28 (127.2.235.252:40805)
I20260620 20:18:37.727151 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.726674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57710)
0620 20:18:37.726928 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.726998 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":64,"threads_started":1}
W20260620 20:18:37.727327 9680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.727598 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.723222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39357 (local address 127.0.0.1:55198)
0620 20:18:37.723394 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.723407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:37.723518 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:37.723903 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:37.723906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:37.723931 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:37.724082 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.724090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.725131 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.725134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.726121 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.726131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.727177 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.727183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.727295 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.727318 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:18:37.727386 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:18:37.727477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":52,"threads_started":1}
I20260620 20:18:37.727634 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.723366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.2.235.253:39357)
0620 20:18:37.723633 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:37.723638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:37.723651 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:37.723696 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:37.723698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:37.723739 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:37.723826 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:37.724227 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.725007 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.726239 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.726879 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.726993 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.727088 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:37.727419 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 20:18:37.727496 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":61,"threads_started":1}
I20260620 20:18:37.728034 9703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09220ecc32d8489dbaf47934d329bb28" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" is_pre_election: true
I20260620 20:18:37.728173 9703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09220ecc32d8489dbaf47934d329bb28 in term 1.
I20260620 20:18:37.728410 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.728150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57720)
0620 20:18:37.728316 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.728361 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 20:18:37.728415 9873 leader_election.cc:302] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09220ecc32d8489dbaf47934d329bb28, 1f79b50b6fdb46388c9e8d5cb38ba4fd; no voters: 33ff3378d82e43a0894307e92d2a1997
I20260620 20:18:37.728591 10408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260620 20:18:37.728585 9680 leader_election.cc:334] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099): Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.728657 10408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 33ff3378d82e43a0894307e92d2a1997)
I20260620 20:18:37.728701 10408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:18:37.729794 10408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } }
I20260620 20:18:37.730113 10408 leader_election.cc:288] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 2 election: Requested vote from peers 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099), 1f79b50b6fdb46388c9e8d5cb38ba4fd (127.2.235.253:39357)
I20260620 20:18:37.730448 9703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09220ecc32d8489dbaf47934d329bb28" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd"
I20260620 20:18:37.730497 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.730236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57726)
0620 20:18:37.730385 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.730432 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 20:18:37.730556 9703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:18:37.730926 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.726694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:40805 (local address 127.0.0.1:37708)
0620 20:18:37.727040 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.727053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:37.727140 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:37.727821 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:37.727825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:37.727847 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:37.727921 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.727926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.728907 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.728909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.729679 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.729687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.730538 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.730541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.730641 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.730657 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 20:18:37.730742 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:18:37.730809 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":69,"threads_started":1}
I20260620 20:18:37.730937 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.726953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.2.235.252:40805)
0620 20:18:37.727214 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:37.727607 (+ 393us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:37.727624 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:18:37.727663 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:37.727666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:37.727707 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:37.727792 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:37.728040 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.728789 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.729800 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.730407 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.730538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.730590 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:37.730762 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:18:37.730829 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260620 20:18:37.731315 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.731078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57740)
0620 20:18:37.731213 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.731254 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 20:18:37.731384 9874 leader_election.cc:334] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099): Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
I20260620 20:18:37.731384 9897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "09220ecc32d8489dbaf47934d329bb28" is_pre_election: true
I20260620 20:18:37.731616 9897 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f79b50b6fdb46388c9e8d5cb38ba4fd in current term 2: Already voted for candidate 09220ecc32d8489dbaf47934d329bb28 in this term.
I20260620 20:18:37.731777 9703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09220ecc32d8489dbaf47934d329bb28 in term 2.
I20260620 20:18:37.732059 9678 leader_election.cc:302] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1f79b50b6fdb46388c9e8d5cb38ba4fd; no voters: 09220ecc32d8489dbaf47934d329bb28, 33ff3378d82e43a0894307e92d2a1997
I20260620 20:18:37.732108 9873 leader_election.cc:302] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09220ecc32d8489dbaf47934d329bb28, 1f79b50b6fdb46388c9e8d5cb38ba4fd; no voters: 33ff3378d82e43a0894307e92d2a1997
I20260620 20:18:37.732249 10413 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:18:37.732343 10408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:18:37.732542 10408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [term 2 LEADER]: Becoming Leader. State: Replica: 09220ecc32d8489dbaf47934d329bb28, State: Running, Role: LEADER
I20260620 20:18:37.732833 10408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } }
I20260620 20:18:37.733879 10418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09220ecc32d8489dbaf47934d329bb28. Latest consensus state: current_term: 2 leader_uuid: "09220ecc32d8489dbaf47934d329bb28" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:37.734004 10418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:37.734238 10420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:37.736171 10420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:37.736567 10420 catalog_manager.cc:1259] Loaded cluster ID: 6a746e6d0f4341209871a0b8182d8522
I20260620 20:18:37.736632 10420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:37.737180 10420 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:37.737565 10420 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Loaded TSK: 0
I20260620 20:18:37.738281 10420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:37.743893 9886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:37.746723 9886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:37.749064 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.748860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57768)
0620 20:18:37.748944 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.749015 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 20:18:37.749125 9703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [term 2 FOLLOWER]: Refusing update from remote peer 09220ecc32d8489dbaf47934d329bb28: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 20:18:37.749563 10418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 20:18:37.749670 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.749489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.0.0.1:57770)
0620 20:18:37.749584 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.749625 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:18:37.749812 9874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 -> Peer 33ff3378d82e43a0894307e92d2a1997 (127.2.235.254:34099): Couldn't send request to peer 33ff3378d82e43a0894307e92d2a1997. Status: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:18:37.750694 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09220ecc32d8489dbaf47934d329bb28. Latest consensus state: current_term: 2 leader_uuid: "09220ecc32d8489dbaf47934d329bb28" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:37.750793 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:37.751742 10408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "09220ecc32d8489dbaf47934d329bb28" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:37.751840 10408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:37.752128 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "09220ecc32d8489dbaf47934d329bb28" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "33ff3378d82e43a0894307e92d2a1997" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 34099 } } peers { permanent_uuid: "1f79b50b6fdb46388c9e8d5cb38ba4fd" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "09220ecc32d8489dbaf47934d329bb28" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 40805 } attrs { promote: false } } }
I20260620 20:18:37.752218 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f79b50b6fdb46388c9e8d5cb38ba4fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:37.762219 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.757934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:39229 (local address 127.0.0.1:43812)
0620 20:18:37.758178 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.758192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:37.758289 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:37.758743 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:37.758749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:37.758790 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:37.758912 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.758919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.760059 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.760062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.761161 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.761169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.761852 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.761855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.761945 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.761977 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:18:37.762062 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:18:37.762120 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":20}
I20260620 20:18:37.762259 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.758064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.2.235.194:39229)
0620 20:18:37.758272 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:37.758277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:37.758292 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:18:37.758375 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:37.758380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:37.758610 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:37.759052 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:37.759156 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.759922 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.761314 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.761725 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.761842 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.761897 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:37.762075 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 20:18:37.762140 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260620 20:18:37.762579 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.757219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40047 (local address 127.0.0.1:44180)
0620 20:18:37.757354 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.757366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:37.757431 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:37.758896 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:37.758899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:37.758914 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:37.759033 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.759041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.760025 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.760028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.761216 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.761223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.762133 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.762137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.762260 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.762299 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:18:37.762360 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 20:18:37.762434 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":85}
I20260620 20:18:37.762740 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.757955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40531 (local address 127.0.0.1:35646)
0620 20:18:37.758254 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.758276 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:37.758353 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:37.758752 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:37.758755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:37.758772 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:37.758884 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.758890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.759843 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.759849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.761192 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:37.761203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.762133 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.762137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:37.762265 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.762428 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 20:18:37.762484 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 20:18:37.762560 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":218,"spinlock_wait_cycles":9088,"thread_start_us":107,"threads_started":1}
I20260620 20:18:37.762826 10137 tablet_service.cc:1467] Processing CreateTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 (DEFAULT_TABLE table=test-table [id=7fb4677eb89f442ca5508a6da471a802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:37.763095 10137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df57864c1e94f99a851c3f32304a4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:37.763129 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.757357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.2.235.195:40047)
0620 20:18:37.758587 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:37.758591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:37.758604 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:37.758652 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:37.758657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:37.758728 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:37.758894 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:37.759128 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.759922 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.761370 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.761997 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.762781 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.762850 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:37.762892 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 20:18:37.762981 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":73,"threads_started":1}
I20260620 20:18:37.763314 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.758307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.2.235.193:40531)
0620 20:18:37.758524 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:37.758527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:37.758539 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:37.758584 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:37.758586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:37.758623 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:37.758742 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:37.759063 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.759731 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.761380 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.761997 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.762947 (+ 950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.763032 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:37.763068 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 20:18:37.763150 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260620 20:18:37.763845 10000 tablet_service.cc:1467] Processing CreateTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 (DEFAULT_TABLE table=test-table [id=7fb4677eb89f442ca5508a6da471a802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:37.764142 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df57864c1e94f99a851c3f32304a4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:37.765321 10431 tablet_bootstrap.cc:492] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Bootstrap starting.
I20260620 20:18:37.765677 10273 tablet_service.cc:1467] Processing CreateTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 (DEFAULT_TABLE table=test-table [id=7fb4677eb89f442ca5508a6da471a802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:37.765969 10273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df57864c1e94f99a851c3f32304a4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:37.766523 10432 tablet_bootstrap.cc:492] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8: Bootstrap starting.
I20260620 20:18:37.766549 10431 tablet_bootstrap.cc:654] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:37.767359 10432 tablet_bootstrap.cc:654] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:37.768250 10435 tablet_bootstrap.cc:492] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Bootstrap starting.
I20260620 20:18:37.768491 10432 tablet_bootstrap.cc:492] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8: No bootstrap required, opened a new log
I20260620 20:18:37.768580 10432 ts_tablet_manager.cc:1397] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:37.768795 10431 tablet_bootstrap.cc:492] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: No bootstrap required, opened a new log
I20260620 20:18:37.768885 10431 ts_tablet_manager.cc:1397] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 20:18:37.769038 10432 raft_consensus.cc:348] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.769130 10435 tablet_bootstrap.cc:654] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:37.769160 10432 raft_consensus.cc:374] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:37.769192 10432 raft_consensus.cc:729] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df947cfb30144ad88d70d12a298746f8, State: Initialized, Role: FOLLOWER
I20260620 20:18:37.769342 10432 consensus_queue.cc:260] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.769346 10431 raft_consensus.cc:348] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.769450 10431 raft_consensus.cc:374] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:37.769481 10431 raft_consensus.cc:729] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7599a977b004257a29b1f305af05e8f, State: Initialized, Role: FOLLOWER
I20260620 20:18:37.769600 10431 consensus_queue.cc:260] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.769701 10432 ts_tablet_manager.cc:1428] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:37.769874 10431 ts_tablet_manager.cc:1428] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:37.770493 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.770125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:57125)
0620 20:18:37.770360 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.770420 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260620 20:18:37.771256 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.770855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.194:53121)
0620 20:18:37.771085 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.771143 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260620 20:18:37.772505 10203 heartbeater.cc:503] Master 127.2.235.252:40805 was elected leader, sending a full tablet report...
I20260620 20:18:37.772655 10066 heartbeater.cc:503] Master 127.2.235.252:40805 was elected leader, sending a full tablet report...
I20260620 20:18:37.773316 10435 tablet_bootstrap.cc:492] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: No bootstrap required, opened a new log
I20260620 20:18:37.773343 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.773095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:58219)
0620 20:18:37.773210 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.773280 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:18:37.773420 10435 ts_tablet_manager.cc:1397] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 20:18:37.773906 10435 raft_consensus.cc:348] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.774015 10435 raft_consensus.cc:374] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:37.774049 10435 raft_consensus.cc:729] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b39446a91e14c1e8c0c01e650194f95, State: Initialized, Role: FOLLOWER
I20260620 20:18:37.774160 10435 consensus_queue.cc:260] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.774523 10435 ts_tablet_manager.cc:1428] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:37.775496 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.774749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:53669)
0620 20:18:37.775369 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.775422 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":78,"threads_started":1}
I20260620 20:18:37.776192 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.775944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:42137)
0620 20:18:37.776081 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.776132 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:18:37.776286 10339 heartbeater.cc:503] Master 127.2.235.252:40805 was elected leader, sending a full tablet report...
W20260620 20:18:37.804713 10342 tablet.cc:2369] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:37.847000 10436 raft_consensus.cc:482] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:37.847107 10436 raft_consensus.cc:504] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.847594 10436 leader_election.cc:288] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047), a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:37.847940 10293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df57864c1e94f99a851c3f32304a4b7" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39446a91e14c1e8c0c01e650194f95" is_pre_election: true
I20260620 20:18:37.848019 10157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df57864c1e94f99a851c3f32304a4b7" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7599a977b004257a29b1f305af05e8f" is_pre_election: true
I20260620 20:18:37.848094 10293 raft_consensus.cc:2454] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 0.
I20260620 20:18:37.848201 10157 raft_consensus.cc:2454] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 0.
I20260620 20:18:37.848470 9955 leader_election.cc:302] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39446a91e14c1e8c0c01e650194f95, df947cfb30144ad88d70d12a298746f8; no voters:
I20260620 20:18:37.848652 10436 raft_consensus.cc:2781] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:37.848712 10436 raft_consensus.cc:482] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:37.848737 10436 raft_consensus.cc:3037] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:37.849694 10436 raft_consensus.cc:504] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.850026 10436 leader_election.cc:288] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 election: Requested vote from peers 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047), a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229)
I20260620 20:18:37.850347 10157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df57864c1e94f99a851c3f32304a4b7" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7599a977b004257a29b1f305af05e8f"
I20260620 20:18:37.850363 10293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df57864c1e94f99a851c3f32304a4b7" candidate_uuid: "df947cfb30144ad88d70d12a298746f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b39446a91e14c1e8c0c01e650194f95"
I20260620 20:18:37.850467 10157 raft_consensus.cc:3037] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:37.850517 10293 raft_consensus.cc:3037] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:37.851362 10293 raft_consensus.cc:2454] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 1.
I20260620 20:18:37.851444 10157 raft_consensus.cc:2454] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df947cfb30144ad88d70d12a298746f8 in term 1.
I20260620 20:18:37.851646 9955 leader_election.cc:302] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b39446a91e14c1e8c0c01e650194f95, df947cfb30144ad88d70d12a298746f8; no voters:
I20260620 20:18:37.851826 10436 raft_consensus.cc:2781] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:37.852005 10436 raft_consensus.cc:686] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [term 1 LEADER]: Becoming Leader. State: Replica: df947cfb30144ad88d70d12a298746f8, State: Running, Role: LEADER
I20260620 20:18:37.852146 10436 consensus_queue.cc:237] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } }
I20260620 20:18:37.852727 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.852449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:37949)
0620 20:18:37.852592 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.852647 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:18:37.854254 9887 catalog_manager.cc:5495] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df947cfb30144ad88d70d12a298746f8 (127.2.235.193). New cstate: current_term: 1 leader_uuid: "df947cfb30144ad88d70d12a298746f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df947cfb30144ad88d70d12a298746f8" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40531 } health_report { overall_health: HEALTHY } } }
I20260620 20:18:37.883104 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.880657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.2.235.193:40531)
0620 20:18:37.880770 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:37.880774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:37.880791 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:18:37.880899 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:37.880901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:37.880936 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:37.880991 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:37.881164 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.881634 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.882412 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:37.882591 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:37.882665 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:37.882836 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:37.882888 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:37.882933 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:37.882997 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:18:37.883031 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:18:37.884735 10293 raft_consensus.cc:1261] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Refusing update from remote peer df947cfb30144ad88d70d12a298746f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:37.884747 10157 raft_consensus.cc:1261] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Refusing update from remote peer df947cfb30144ad88d70d12a298746f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:37.885140 10443 consensus_queue.cc:1035] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7599a977b004257a29b1f305af05e8f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:37.885319 10436 consensus_queue.cc:1035] T 8df57864c1e94f99a851c3f32304a4b7 P df947cfb30144ad88d70d12a298746f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b39446a91e14c1e8c0c01e650194f95" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:37.887668 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.887424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.194:59931)
0620 20:18:37.887561 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.887612 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 20:18:37.887845 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.887598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:36739)
0620 20:18:37.887738 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.887785 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:18:37.889119 10446 mvcc.cc:205] Tried to move back new op lower bound from 7299017596452741120 to 7299017596325404672. Current Snapshot: MvccSnapshot[applied={T|T < 7299017596452741120}]
I20260620 20:18:37.890161 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.889936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:60257)
0620 20:18:37.890061 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.890104 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 20:18:37.891618 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.891414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.194:48251)
0620 20:18:37.891524 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.891566 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:18:37.893623 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.893394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.193:47195)
0620 20:18:37.893505 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.893568 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:18:37.893097 10448 mvcc.cc:205] Tried to move back new op lower bound from 7299017596452741120 to 7299017596325404672. Current Snapshot: MvccSnapshot[applied={T|T < 7299017596452741120 or (T in {7299017596452741120})}]
I20260620 20:18:37.894445 9886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:37.894583 9886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:37.895201 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.894987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:34099 (local address 127.2.235.195:60201)
0620 20:18:37.895100 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.895145 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 20:18:37.898351 9886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09220ecc32d8489dbaf47934d329bb28: Sending DeleteTablet for 3 replicas of tablet 8df57864c1e94f99a851c3f32304a4b7
I20260620 20:18:37.898926 10273 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:37 UTC) from {username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:44180
I20260620 20:18:37.899005 10137 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:37 UTC) from {username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:43812
I20260620 20:18:37.899160 10272 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:37 UTC) from {username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:44180
I20260620 20:18:37.899219 10000 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df57864c1e94f99a851c3f32304a4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:37 UTC) from {username='oryx', principal='oryx/127.2.235.252@KRBTEST.COM'} at 127.0.0.1:35646
I20260620 20:18:37.899317 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9840
I20260620 20:18:37.900275 10455 tablet_replica.cc:330] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: stopping tablet replica
I20260620 20:18:37.900413 10455 raft_consensus.cc:2229] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:37.900529 10455 raft_consensus.cc:2258] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:37.901017 10455 ts_tablet_manager.cc:1905] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:37.902814 10455 ts_tablet_manager.cc:1918] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:37.902870 10455 log.cc:1196] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/8df57864c1e94f99a851c3f32304a4b7
I20260620 20:18:37.903095 10455 ts_tablet_manager.cc:1939] T 8df57864c1e94f99a851c3f32304a4b7 P a7599a977b004257a29b1f305af05e8f: Deleting consensus metadata
I20260620 20:18:37.903645 9872 catalog_manager.cc:4841] TS a7599a977b004257a29b1f305af05e8f (127.2.235.194:39229): tablet 8df57864c1e94f99a851c3f32304a4b7 (table test-table [id=7fb4677eb89f442ca5508a6da471a802]) successfully deleted
I20260620 20:18:37.905375 10454 tablet_replica.cc:330] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: stopping tablet replica
I20260620 20:18:37.905493 10454 raft_consensus.cc:2229] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:37.905614 10454 raft_consensus.cc:2258] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:37.906193 10454 ts_tablet_manager.cc:1905] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:37.906637 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 10074
I20260620 20:18:37.906816 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:37.906536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40531 (local address 127.0.0.1:35664)
0620 20:18:37.906662 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:37.906724 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:18:37.907572 9871 catalog_manager.cc:4584] TS df947cfb30144ad88d70d12a298746f8 (127.2.235.193:40531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8df57864c1e94f99a851c3f32304a4b7: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40531: connect: Connection refused (error 111)
I20260620 20:18:37.908650 10454 ts_tablet_manager.cc:1918] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:37.908735 10454 log.cc:1196] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/8df57864c1e94f99a851c3f32304a4b7
I20260620 20:18:37.908973 10454 ts_tablet_manager.cc:1939] T 8df57864c1e94f99a851c3f32304a4b7 P 8b39446a91e14c1e8c0c01e650194f95: Deleting consensus metadata
I20260620 20:18:37.909614 9873 catalog_manager.cc:4841] TS 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047): tablet 8df57864c1e94f99a851c3f32304a4b7 (table test-table [id=7fb4677eb89f442ca5508a6da471a802]) successfully deleted
W20260620 20:18:37.909932 9873 catalog_manager.cc:4808] TS 8b39446a91e14c1e8c0c01e650194f95 (127.2.235.195:40047): delete failed for tablet 8df57864c1e94f99a851c3f32304a4b7 because the tablet was not found. No further retry: Not found: Tablet not found: 8df57864c1e94f99a851c3f32304a4b7
I20260620 20:18:37.912848 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 10210
I20260620 20:18:37.918269 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9560
I20260620 20:18:37.923595 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 9747
2026-06-20T20:18:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5007 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:37 dist-test-slave-8z53 krb5kdc[10459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:37 dist-test-slave-8z53 krb5kdc[10459](info): set up 2 sockets
Jun 20 20:18:37 dist-test-slave-8z53 krb5kdc[10459](info): commencing operation
krb5kdc: starting...
W20260620 20:18:38.885723 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:38 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab.
Entry for principal oryx/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab.
I20260620 20:18:38.991950 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33685
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab
--principal=oryx/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:33685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:39.090829 10472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:39.091039 10472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:39.091077 10472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:39.094686 10472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:39.094734 10472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:18:39.094753 10472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:39.094820 10472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:39.094851 10472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:39.094890 10472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:39.099655 10472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:33685
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.254.keytab
--principal=oryx/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:39.100456 10472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:39.101363 10472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:39.108832 10472 server_base.cc:1034] running on GCE node
W20260620 20:18:39.108747 10477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.108778 10478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.108762 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:39.109479 10472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:39.110308 10472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:39.111457 10472 hybrid_clock.cc:648] HybridClock initialized: now 1781986719111440 us; error 30 us; skew 500 ppm
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:39.113735 10472 init.cc:377] Logged in from keytab as oryx/127.2.235.254@KRBTEST.COM (short username oryx)
I20260620 20:18:39.114420 10472 webserver.cc:460] Webserver started at http://127.2.235.254:35521/ using document root <none> and password file <none>
I20260620 20:18:39.114835 10472 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:39.114883 10472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:39.115013 10472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:39.116698 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "ffa09ff7e83847a7a289394b379175a9"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "31d7346a5772cf6ea1dcd539bb3715fa"
server_key_iv: "440e27fc8f55b058e284ad761f610e8c"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.117152 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "ffa09ff7e83847a7a289394b379175a9"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "31d7346a5772cf6ea1dcd539bb3715fa"
server_key_iv: "440e27fc8f55b058e284ad761f610e8c"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.120235 10472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:18:39.122308 10487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.123416 10472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:39.123531 10472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "ffa09ff7e83847a7a289394b379175a9"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "31d7346a5772cf6ea1dcd539bb3715fa"
server_key_iv: "440e27fc8f55b058e284ad761f610e8c"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.123624 10472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:39.139158 10472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:39.141732 10472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:39.141909 10472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:39.148778 10472 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:33685
I20260620 20:18:39.148790 10549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:33685 every 8 connection(s)
I20260620 20:18:39.149614 10472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:39.152096 10550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:39.156332 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 10472
I20260620 20:18:39.156438 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:39.156751 2991 external_mini_cluster.cc:1428] Setting key 1bfd1e407d58e5448bf6ff13911d3fd0
I20260620 20:18:39.157058 10550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Bootstrap starting.
I20260620 20:18:39.159348 10550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:39.160156 10550 log.cc:824] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:39.162161 10550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: No bootstrap required, opened a new log
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:39.165776 10550 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa09ff7e83847a7a289394b379175a9" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33685 } }
I20260620 20:18:39.166004 10550 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:39.166106 10550 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa09ff7e83847a7a289394b379175a9, State: Initialized, Role: FOLLOWER
I20260620 20:18:39.166597 10550 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa09ff7e83847a7a289394b379175a9" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33685 } }
I20260620 20:18:39.166766 10550 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:39.166841 10550 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:39.166947 10550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:39.168011 10550 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa09ff7e83847a7a289394b379175a9" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33685 } }
I20260620 20:18:39.168380 10550 leader_election.cc:302] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa09ff7e83847a7a289394b379175a9; no voters:
I20260620 20:18:39.168735 10550 leader_election.cc:288] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:39.168823 10555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:39.169055 10555 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [term 1 LEADER]: Becoming Leader. State: Replica: ffa09ff7e83847a7a289394b379175a9, State: Running, Role: LEADER
I20260620 20:18:39.169370 10555 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa09ff7e83847a7a289394b379175a9" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33685 } }
I20260620 20:18:39.169809 10550 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:39.171734 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.157998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.2.235.254:33685)
0620 20:18:39.158402 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.158415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.158444 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:18:39.159018 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.159033 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.159333 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.159752 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.160682 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.161989 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.162930 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.163653 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.163824 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.166230 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.166236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.166251 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.166279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.168248 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.168731 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.168737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.168744 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.168882 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.169228 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.169231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.169233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.169697 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.169866 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.170505 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.170828 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":146,"threads_started":1}
I20260620 20:18:39.172439 10556 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa09ff7e83847a7a289394b379175a9. Latest consensus state: current_term: 1 leader_uuid: "ffa09ff7e83847a7a289394b379175a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa09ff7e83847a7a289394b379175a9" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33685 } } }
I20260620 20:18:39.172493 10557 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa09ff7e83847a7a289394b379175a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa09ff7e83847a7a289394b379175a9" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33685 } } }
I20260620 20:18:39.172830 10557 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:39.172828 10556 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9 [sys.catalog]: This master's current role is: LEADER
W20260620 20:18:39.174729 10569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:39.174800 10569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:39.174849 10571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:39.175666 10571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:39.180414 10571 catalog_manager.cc:1347] Generated new cluster ID: 11b33ab3ae664c3ea3cf657e48d539da
I20260620 20:18:39.180472 10571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:39.200464 10571 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:39.201458 10571 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:39.207011 10571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Generated new TSK 0
I20260620 20:18:39.207661 10571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab.
Entry for principal oryx/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab.
I20260620 20:18:39.233889 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33685
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:39.332438 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:39.332652 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:39.332744 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:39.336555 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:39.336601 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:39.336638 10576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:39.336714 10576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:39.341465 10576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:33685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:39.342272 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:39.343067 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:39.351369 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.351377 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.351388 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:39.351737 10576 server_base.cc:1034] running on GCE node
I20260620 20:18:39.352131 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:39.352674 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:39.353827 10576 hybrid_clock.cc:648] HybridClock initialized: now 1781986719353793 us; error 45 us; skew 500 ppm
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:39.356232 10576 init.cc:377] Logged in from keytab as oryx/127.2.235.193@KRBTEST.COM (short username oryx)
I20260620 20:18:39.356993 10576 webserver.cc:460] Webserver started at http://127.2.235.193:35089/ using document root <none> and password file <none>
I20260620 20:18:39.357558 10576 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:39.357652 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:39.357905 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:39.359680 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "1ccd43d49d674268aa3ab147cd37d9cc"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "3d239adeb0bf5535f8aa9d321daaa594"
server_key_iv: "d864e23ccc623a7bf6778a88e917bfb8"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.360320 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "1ccd43d49d674268aa3ab147cd37d9cc"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "3d239adeb0bf5535f8aa9d321daaa594"
server_key_iv: "d864e23ccc623a7bf6778a88e917bfb8"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.363584 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 20:18:39.365653 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.366742 10576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 20:18:39.366873 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "1ccd43d49d674268aa3ab147cd37d9cc"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "3d239adeb0bf5535f8aa9d321daaa594"
server_key_iv: "d864e23ccc623a7bf6778a88e917bfb8"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.366972 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:39.381363 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:39.383683 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:39.383867 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:39.385027 10576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:39.385095 10576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.385154 10576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:39.385201 10576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:39.394842 10576 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:43939
I20260620 20:18:39.395273 10706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:43939 every 8 connection(s)
I20260620 20:18:39.395725 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:39.398537 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.385676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:45809 (local address 127.2.235.254:33685)
0620 20:18:39.385811 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.385816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.386607 (+ 791us) server_negotiation.cc:407] Connection header received
0620 20:18:39.387623 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.387628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.387690 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.387749 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.389352 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.390162 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.391757 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.392132 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.392254 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.394848 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.394851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.394853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.394878 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.396593 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.397066 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.397071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.397073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.397155 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.397511 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.397514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.397516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.397684 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.397808 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.398255 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.398367 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:18:39.398540 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 10576
I20260620 20:18:39.398676 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:39.398957 2991 external_mini_cluster.cc:1428] Setting key 1709b0f49a957f1fd280b71837808fbe
I20260620 20:18:39.399209 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.386021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.2.235.193:45809)
0620 20:18:39.386446 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.386490 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.387384 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.388127 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.388135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.388586 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:39.389116 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.389133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.390307 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.390311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.391627 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.391638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.392278 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.392281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.392415 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.392958 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:39.392978 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:39.394681 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:39.396703 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.396709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.396722 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.396966 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.397254 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.397275 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.397277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.397411 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.397795 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:39.397801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:39.398018 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.398188 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.398473 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":130,"threads_started":1}
I20260620 20:18:39.400429 10707 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33685
I20260620 20:18:39.400563 10707 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:39.401075 10707 heartbeater.cc:511] Master 127.2.235.254:33685 requested a full tablet report, sending...
I20260620 20:18:39.402487 10502 ts_manager.cc:194] Registered new tserver with Master: 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939)
I20260620 20:18:39.403899 10502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:45809
WARNING: no policy specified for oryx/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:18:39.409960 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.402230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.2.235.193:51433)
0620 20:18:39.402995 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.403005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.403103 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.403301 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.403304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.403476 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:39.403553 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.403557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.404282 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.404285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.405792 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.405800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.406216 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.406220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.406307 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.406781 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:39.406792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:39.407068 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:39.408536 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.408540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.408542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.408718 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.408960 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.408963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.408965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.409037 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.409459 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:39.409461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:39.409546 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.409650 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.409796 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":103,"threads_started":1}
I20260620 20:18:39.409977 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.402276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:51433 (local address 127.2.235.254:33685)
0620 20:18:39.402358 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.402362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.403073 (+ 711us) server_negotiation.cc:407] Connection header received
0620 20:18:39.403170 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.403173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.403206 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.403288 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.403665 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.404179 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.405900 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.406111 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.406220 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.407194 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.407198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.407200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.407231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.408448 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.408804 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.408808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.408809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.408878 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.409157 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.409160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.409161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.409374 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.409465 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.409697 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.409822 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab.
Entry for principal oryx/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab.
I20260620 20:18:39.423437 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33685
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:39.524840 10716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:39.525048 10716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:39.525089 10716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:39.528971 10716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:39.529034 10716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:39.529071 10716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:39.529101 10716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:39.533740 10716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:33685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:39.534459 10716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:39.535189 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:39.543407 10721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.543464 10724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.543481 10722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:39.543686 10716 server_base.cc:1034] running on GCE node
I20260620 20:18:39.544031 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:39.544531 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:39.545696 10716 hybrid_clock.cc:648] HybridClock initialized: now 1781986719545695 us; error 37 us; skew 500 ppm
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:39.548521 10716 init.cc:377] Logged in from keytab as oryx/127.2.235.194@KRBTEST.COM (short username oryx)
I20260620 20:18:39.549639 10716 webserver.cc:460] Webserver started at http://127.2.235.194:46059/ using document root <none> and password file <none>
I20260620 20:18:39.550345 10716 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:39.550472 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:39.550765 10716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:39.553526 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "705fb25854e8ad01c25c23f864c3ff19"
server_key_iv: "67b1d86933043fc1011332788e796753"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.554167 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "705fb25854e8ad01c25c23f864c3ff19"
server_key_iv: "67b1d86933043fc1011332788e796753"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.558115 10716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:18:39.565642 10731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.567108 10716 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260620 20:18:39.567242 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "705fb25854e8ad01c25c23f864c3ff19"
server_key_iv: "67b1d86933043fc1011332788e796753"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.567389 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:39.583630 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:39.587329 10716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:39.587531 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:39.589447 10716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:39.589526 10716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.589594 10716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:39.589630 10716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:39.601900 10716 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:35539
I20260620 20:18:39.602020 10846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:35539 every 8 connection(s)
I20260620 20:18:39.603039 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:39.603197 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.589785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:38249 (local address 127.2.235.254:33685)
0620 20:18:39.589920 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.589925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.590645 (+ 720us) server_negotiation.cc:407] Connection header received
0620 20:18:39.591555 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.591558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.591609 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.591708 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.592938 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.593736 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.595326 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.595670 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.595804 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.599284 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.599289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.599295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.599324 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.601023 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.601612 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.601616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.601618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.601704 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.602160 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.602166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.602168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.602341 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.602446 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.602915 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.603027 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:18:39.604068 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.590111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.2.235.194:38249)
0620 20:18:39.590519 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.590562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.591354 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.591868 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.591874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.592233 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:39.592742 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.592775 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.593845 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.593848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.595169 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.595182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.595832 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.595837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.596021 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.596645 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:39.596668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:39.599111 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:39.601124 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.601129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.601144 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.601486 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.601855 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.601860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.601864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.602045 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.602454 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:39.602462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:39.602684 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.602860 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.603151 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":128,"threads_started":1}
I20260620 20:18:39.605284 10847 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33685
I20260620 20:18:39.605418 10847 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:39.605908 10847 heartbeater.cc:511] Master 127.2.235.254:33685 requested a full tablet report, sending...
I20260620 20:18:39.607040 10502 ts_manager.cc:194] Registered new tserver with Master: 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539)
I20260620 20:18:39.607867 10502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:38249
I20260620 20:18:39.608582 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 10716
I20260620 20:18:39.608709 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:39.608994 2991 external_mini_cluster.cc:1428] Setting key 5a7598727ec2872be87609d24ee9d533
WARNING: no policy specified for oryx/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:18:39.618824 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.607473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:40995 (local address 127.2.235.254:33685)
0620 20:18:39.607567 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.607571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.607764 (+ 193us) server_negotiation.cc:407] Connection header received
0620 20:18:39.607862 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.607865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.607907 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.607972 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.609333 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.609913 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.612958 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.613371 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.613507 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.614909 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.614912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.614915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.614941 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.616657 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.617218 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.617224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.617227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.617354 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.617699 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.617702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.617706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.617971 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.618047 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.618479 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.618624 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:18:39.619410 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.607419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.2.235.194:40995)
0620 20:18:39.607686 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.607695 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.607796 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.608019 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.608022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.608210 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:39.608311 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.608317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.610030 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.610033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.612824 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.612838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.613637 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.613643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.613764 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.614325 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:39.614343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:39.614768 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:39.616803 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.616808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.616810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.617081 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.617486 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.617491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.617495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.617589 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.618120 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:39.618124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:39.618254 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.618985 (+ 731us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.619143 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab.
Entry for principal oryx/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab.
I20260620 20:18:39.633963 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33685
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:39.735646 10856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:39.735893 10856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:39.735971 10856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:39.739809 10856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:39.739888 10856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:39.739970 10856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:18:39.740051 10856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:39.745056 10856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.235.254:33685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:39.745893 10856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:39.746788 10856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:39.755630 10862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:39.755682 10864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:39.755793 10856 server_base.cc:1034] running on GCE node
W20260620 20:18:39.755694 10861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:39.756310 10856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:39.756867 10856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:39.758038 10856 hybrid_clock.cc:648] HybridClock initialized: now 1781986719758022 us; error 27 us; skew 500 ppm
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:39.760457 10856 init.cc:377] Logged in from keytab as oryx/127.2.235.195@KRBTEST.COM (short username oryx)
I20260620 20:18:39.761341 10856 webserver.cc:460] Webserver started at http://127.2.235.195:38187/ using document root <none> and password file <none>
I20260620 20:18:39.761849 10856 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:39.761929 10856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:39.762147 10856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:39.763935 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "5c8e0053952044f287ff0391e3155661"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "5fb1e8c227a3a9b5cb42eccf83b7e004"
server_key_iv: "88aec34680e36058e4568bff19e1a5f3"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.764441 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "5c8e0053952044f287ff0391e3155661"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "5fb1e8c227a3a9b5cb42eccf83b7e004"
server_key_iv: "88aec34680e36058e4568bff19e1a5f3"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.767788 10856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:18:39.769958 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.771123 10856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:18:39.771235 10856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "5c8e0053952044f287ff0391e3155661"
format_stamp: "Formatted at 2026-06-20 20:18:39 on dist-test-slave-8z53"
server_key: "5fb1e8c227a3a9b5cb42eccf83b7e004"
server_key_iv: "88aec34680e36058e4568bff19e1a5f3"
server_key_version: "encryptionkey@0"
I20260620 20:18:39.771338 10856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:39.784708 10856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:39.787299 10856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:39.787478 10856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:39.789203 10856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:39.789289 10856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:39.789342 10856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:39.789369 10856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:39.799649 10856 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:43155
I20260620 20:18:39.799820 10986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:43155 every 8 connection(s)
I20260620 20:18:39.800684 10856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:39.802438 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.790124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:44187 (local address 127.2.235.254:33685)
0620 20:18:39.790290 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.790295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.790773 (+ 478us) server_negotiation.cc:407] Connection header received
0620 20:18:39.791778 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.791782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.791838 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.791890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.793546 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.794229 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.795322 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.795652 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.795776 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.798725 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.798728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.798731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.798756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.800416 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.800927 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.800931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.800933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.801025 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.801435 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.801438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.801440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.801666 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.801747 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.802212 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.802323 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:39.803066 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.790167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.2.235.195:44187)
0620 20:18:39.790627 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.790673 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.791536 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.792286 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.792294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.792730 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:39.793398 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.793415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.794347 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.794351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.795194 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.795202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.795831 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.795838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.796078 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.796711 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:39.796732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:39.798545 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:39.800519 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.800524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.800533 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.800830 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.801135 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.801141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.801142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.801325 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.801781 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:39.801785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:39.801991 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.802129 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.802341 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260620 20:18:39.804067 10987 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33685
I20260620 20:18:39.804229 10987 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:39.804648 10987 heartbeater.cc:511] Master 127.2.235.254:33685 requested a full tablet report, sending...
I20260620 20:18:39.805626 10504 ts_manager.cc:194] Registered new tserver with Master: 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:39.806339 10504 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:44187
I20260620 20:18:39.809276 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 10856
I20260620 20:18:39.809384 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:39.809659 2991 external_mini_cluster.cc:1428] Setting key 759bc2e80d89839fe168c6e5a99dca2e
I20260620 20:18:39.811617 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:18:39.812881 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.806126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:58939 (local address 127.2.235.254:33685)
0620 20:18:39.806235 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.806239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.806535 (+ 296us) server_negotiation.cc:407] Connection header received
0620 20:18:39.806636 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.806639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.806685 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.806771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.807215 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.807702 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.808590 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.808791 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.808867 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.809844 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.809848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.809850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.809873 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.811354 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.811810 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.811813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.811815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.811897 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.812167 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.812171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.812173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.812326 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.812422 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.812652 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.812759 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:39.812892 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.806028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.2.235.195:58939)
0620 20:18:39.806403 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.806418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.806559 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.806802 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.806804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.806994 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:39.807101 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.807108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.807825 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.807828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.808465 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.808473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.808882 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.808885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.808945 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.809411 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:39.809428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:39.809732 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:39.811458 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.811461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.811462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.811709 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.811996 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:39.812000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:39.812002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:39.812083 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:39.812414 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:39.812417 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:39.812500 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.812604 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.812731 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":88,"threads_started":1}
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:18:39.833338 10553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:39.827825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.2.235.254:33685)
0620 20:18:39.827922 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.827926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.827954 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:18:39.827988 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.827990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.828019 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.828093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.828655 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.829122 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.830068 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.830279 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.830367 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.833178 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33146: BlockingRecv error: recv got EOF from 127.0.0.1:33146 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 20:18:39 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:39.841320 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.834431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.2.235.254:33685)
0620 20:18:39.834537 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.834541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.834680 (+ 139us) server_negotiation.cc:407] Connection header received
0620 20:18:39.834753 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.834755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.834784 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.834839 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:39.835354 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.835802 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.836524 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.836711 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.836801 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.838788 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:39.838791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:39.838792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:39.838811 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:39.840136 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.840457 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.840459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.840460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.840522 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:39.840715 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:39.840717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:39.840718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:39.840862 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:39.840947 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.841082 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.841151 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:18:39.843727 10504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:39.846570 10504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:39.859738 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.854747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.2.235.194:35539)
0620 20:18:39.855362 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.855375 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.855394 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:18:39.855457 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.855461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.855613 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.855775 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:39.855969 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.856780 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.858533 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.858996 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.859118 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.859298 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.859506 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.859597 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":94,"threads_started":1}
I20260620 20:18:39.859736 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.853980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:35539 (local address 127.0.0.1:58780)
0620 20:18:39.854981 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.854996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.855090 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.855755 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.855758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.855771 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:39.855852 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.855858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.856909 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.856915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.858445 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.858454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.859123 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.859126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.859240 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.859330 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.859477 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.859590 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":231,"thread_start_us":80,"threads_started":1}
I20260620 20:18:39.859795 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.854568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:43155 (local address 127.0.0.1:57298)
0620 20:18:39.855168 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.855182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.855261 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.855913 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.855918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.855937 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:39.856037 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.856043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.857309 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.857317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.858291 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.858299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.859364 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.859367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.859451 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.859497 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.859590 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.859661 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260620 20:18:39.860020 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.854684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.2.235.195:43155)
0620 20:18:39.855063 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.855074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.855344 (+ 270us) server_negotiation.cc:407] Connection header received
0620 20:18:39.855414 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.855421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.855614 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.855775 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:39.856170 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.857169 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.858392 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.859058 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.859159 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.859315 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.859631 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.859711 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":79,"threads_started":1}
I20260620 20:18:39.860267 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.853939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.0.0.1:60558)
0620 20:18:39.854267 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:39.854298 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:39.854378 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:39.855459 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:39.855466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:39.855495 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:39.855571 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.855575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.856970 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.856973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.858375 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:39.858386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.859245 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.859249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:39.859956 (+ 707us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.859985 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:18:39.860072 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:18:39.860145 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":714,"thread_start_us":59,"threads_started":1}
I20260620 20:18:39.860671 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:39.854589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.2.235.193:43939)
0620 20:18:39.854996 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:39.855006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:39.855029 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:18:39.855103 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:39.855108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:39.855302 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:39.855680 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:39.855819 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.856881 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.858493 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:39.859108 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:39.859317 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:39.859449 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:39.860436 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 20:18:39.860526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":94,"threads_started":1}
I20260620 20:18:39.863163 10641 tablet_service.cc:1467] Processing CreateTablet for tablet b9100225d1ce44aeac332d8cf87a1055 (DEFAULT_TABLE table=test-table [id=a98f1bfd770e4d40858f90b819c68b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:39.863162 10781 tablet_service.cc:1467] Processing CreateTablet for tablet b9100225d1ce44aeac332d8cf87a1055 (DEFAULT_TABLE table=test-table [id=a98f1bfd770e4d40858f90b819c68b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:39.864125 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9100225d1ce44aeac332d8cf87a1055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:39.864125 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9100225d1ce44aeac332d8cf87a1055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:39.867218 10910 tablet_service.cc:1467] Processing CreateTablet for tablet b9100225d1ce44aeac332d8cf87a1055 (DEFAULT_TABLE table=test-table [id=a98f1bfd770e4d40858f90b819c68b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:39.868140 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9100225d1ce44aeac332d8cf87a1055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:39.870060 11015 tablet_bootstrap.cc:492] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Bootstrap starting.
I20260620 20:18:39.871455 11016 tablet_bootstrap.cc:492] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Bootstrap starting.
I20260620 20:18:39.872458 11015 tablet_bootstrap.cc:654] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:39.873425 11016 tablet_bootstrap.cc:654] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:39.873451 11015 log.cc:824] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:39.874120 11016 log.cc:824] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:39.875707 11016 tablet_bootstrap.cc:492] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: No bootstrap required, opened a new log
I20260620 20:18:39.875730 11019 tablet_bootstrap.cc:492] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Bootstrap starting.
I20260620 20:18:39.875891 11016 ts_tablet_manager.cc:1397] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:39.877213 11015 tablet_bootstrap.cc:492] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: No bootstrap required, opened a new log
I20260620 20:18:39.877447 11015 ts_tablet_manager.cc:1397] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260620 20:18:39.877615 11019 tablet_bootstrap.cc:654] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:39.878348 11019 log.cc:824] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:39.879622 11016 raft_consensus.cc:348] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:39.879842 11016 raft_consensus.cc:374] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:39.879894 11016 raft_consensus.cc:729] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Initialized, Role: FOLLOWER
I20260620 20:18:39.880180 11019 tablet_bootstrap.cc:492] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: No bootstrap required, opened a new log
I20260620 20:18:39.880357 11019 ts_tablet_manager.cc:1397] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:18:39.880321 11016 consensus_queue.cc:260] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:39.880935 10847 heartbeater.cc:503] Master 127.2.235.254:33685 was elected leader, sending a full tablet report...
I20260620 20:18:39.881233 11016 ts_tablet_manager.cc:1428] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:18:39.882203 11015 raft_consensus.cc:348] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:39.882459 11015 raft_consensus.cc:374] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:39.882519 11015 raft_consensus.cc:729] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Initialized, Role: FOLLOWER
I20260620 20:18:39.882917 11015 consensus_queue.cc:260] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:39.883567 10707 heartbeater.cc:503] Master 127.2.235.254:33685 was elected leader, sending a full tablet report...
I20260620 20:18:39.883819 11015 ts_tablet_manager.cc:1428] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260620 20:18:39.884155 11019 raft_consensus.cc:348] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:39.884400 11019 raft_consensus.cc:374] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:39.884469 11019 raft_consensus.cc:729] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8e0053952044f287ff0391e3155661, State: Initialized, Role: FOLLOWER
I20260620 20:18:39.884836 11019 consensus_queue.cc:260] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:39.885521 10987 heartbeater.cc:503] Master 127.2.235.254:33685 was elected leader, sending a full tablet report...
I20260620 20:18:39.885795 11019 ts_tablet_manager.cc:1428] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 20:18:39.896773 10708 tablet.cc:2369] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:40.015202 11023 raft_consensus.cc:482] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:40.015395 11023 raft_consensus.cc:504] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.016526 11023 leader_election.cc:288] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939), 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539)
I20260620 20:18:40.019999 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.016877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:34213 (local address 127.2.235.194:35539)
0620 20:18:40.017013 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.017017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.017028 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:40.017074 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.017077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.017114 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.017179 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:40.017515 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.018168 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.019086 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.019442 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.019573 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.019620 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.019828 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.019900 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:18:40.019999 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.016744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:35539 (local address 127.2.235.195:34213)
0620 20:18:40.016858 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.016873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.016958 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.017237 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.017240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.017278 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:40.017409 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.017413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.018271 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.018274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.018975 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.018983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.019565 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.019568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.019712 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.019729 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.019812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.019882 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":22}
I20260620 20:18:40.020596 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9100225d1ce44aeac332d8cf87a1055" candidate_uuid: "5c8e0053952044f287ff0391e3155661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" is_pre_election: true
I20260620 20:18:40.020824 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.016803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:36911 (local address 127.2.235.193:43939)
0620 20:18:40.016928 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.016935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.017537 (+ 602us) server_negotiation.cc:407] Connection header received
0620 20:18:40.017642 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.017644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.017680 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.017738 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:40.018048 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.018489 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.019845 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.020341 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.020450 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.020493 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.020661 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.020723 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:18:40.020848 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.016747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.2.235.195:36911)
0620 20:18:40.017428 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.017438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.017545 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.017774 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.017776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.017792 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:40.017900 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.017905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.018611 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.018617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.019703 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.019711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.020450 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.020454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.020543 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.020566 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.020645 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.020706 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":322,"thread_start_us":136,"threads_started":1}
I20260620 20:18:40.020860 10801 raft_consensus.cc:2454] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c8e0053952044f287ff0391e3155661 in term 0.
I20260620 20:18:40.021314 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9100225d1ce44aeac332d8cf87a1055" candidate_uuid: "5c8e0053952044f287ff0391e3155661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" is_pre_election: true
I20260620 20:18:40.021358 10873 leader_election.cc:302] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e0053952044f287ff0391e3155661, 6d77eae8ddfe4c2ebf6294e1f4b389bc; no voters:
I20260620 20:18:40.021588 10661 raft_consensus.cc:2454] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c8e0053952044f287ff0391e3155661 in term 0.
I20260620 20:18:40.021608 11023 raft_consensus.cc:2781] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:40.021677 11023 raft_consensus.cc:482] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:40.021739 11023 raft_consensus.cc:3037] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.022713 11023 raft_consensus.cc:504] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.023037 11023 leader_election.cc:288] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [CANDIDATE]: Term 1 election: Requested vote from peers 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939), 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539)
I20260620 20:18:40.023316 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9100225d1ce44aeac332d8cf87a1055" candidate_uuid: "5c8e0053952044f287ff0391e3155661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc"
I20260620 20:18:40.023319 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9100225d1ce44aeac332d8cf87a1055" candidate_uuid: "5c8e0053952044f287ff0391e3155661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccd43d49d674268aa3ab147cd37d9cc"
I20260620 20:18:40.023424 10801 raft_consensus.cc:3037] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.023425 10661 raft_consensus.cc:3037] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.024602 10661 raft_consensus.cc:2454] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8e0053952044f287ff0391e3155661 in term 1.
I20260620 20:18:40.024906 10872 leader_election.cc:302] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccd43d49d674268aa3ab147cd37d9cc, 5c8e0053952044f287ff0391e3155661; no voters:
I20260620 20:18:40.024958 10801 raft_consensus.cc:2454] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8e0053952044f287ff0391e3155661 in term 1.
I20260620 20:18:40.025118 11023 raft_consensus.cc:2781] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:40.025323 11023 raft_consensus.cc:686] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 1 LEADER]: Becoming Leader. State: Replica: 5c8e0053952044f287ff0391e3155661, State: Running, Role: LEADER
I20260620 20:18:40.025617 11023 consensus_queue.cc:237] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.028504 10502 catalog_manager.cc:5495] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c8e0053952044f287ff0391e3155661 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "5c8e0053952044f287ff0391e3155661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } health_report { overall_health: HEALTHY } } }
I20260620 20:18:40.048672 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.045310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.2.235.195:43155)
0620 20:18:40.045406 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.045409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.045484 (+ 75us) server_negotiation.cc:407] Connection header received
0620 20:18:40.045639 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.045642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.045692 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.045780 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.046005 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.046570 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.047425 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.047629 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.047723 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.047978 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.048356 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.048467 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.048520 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.048563 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:18:40.051674 10988 tablet.cc:2369] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:40.053519 10801 raft_consensus.cc:1261] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Refusing update from remote peer 5c8e0053952044f287ff0391e3155661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.053519 10661 raft_consensus.cc:1261] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Refusing update from remote peer 5c8e0053952044f287ff0391e3155661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.054112 11023 consensus_queue.cc:1035] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.054266 11025 consensus_queue.cc:1035] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.061585 11029 mvcc.cc:205] Tried to move back new op lower bound from 7299017605333422080 to 7299017605227503616. Current Snapshot: MvccSnapshot[applied={T|T < 7299017605333422080}]
I20260620 20:18:40.069166 10504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.069331 10504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.071563 10504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Sending DeleteTablet for 3 replicas of tablet b9100225d1ce44aeac332d8cf87a1055
I20260620 20:18:40.072185 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet b9100225d1ce44aeac332d8cf87a1055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 20:18:40.072213 10910 tablet_service.cc:1514] Processing DeleteTablet for tablet b9100225d1ce44aeac332d8cf87a1055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:57298
I20260620 20:18:40.072294 10641 tablet_service.cc:1514] Processing DeleteTablet for tablet b9100225d1ce44aeac332d8cf87a1055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:60558
I20260620 20:18:40.072590 11039 tablet_replica.cc:330] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: stopping tablet replica
I20260620 20:18:40.072830 11039 raft_consensus.cc:2229] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:40.073004 11040 tablet_replica.cc:330] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: stopping tablet replica
I20260620 20:18:40.073132 11039 raft_consensus.cc:2258] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.073235 11040 raft_consensus.cc:2229] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.073185 10504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:18:40.073421 11038 tablet_replica.cc:330] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: stopping tablet replica
W20260620 20:18:40.073526 10504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:40.073576 11040 raft_consensus.cc:2258] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.074826 11039 ts_tablet_manager.cc:1905] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.074826 11040 ts_tablet_manager.cc:1905] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.075465 11038 raft_consensus.cc:2229] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.075700 11038 raft_consensus.cc:2258] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.078029 11039 ts_tablet_manager.cc:1918] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:40.078109 11039 log.cc:1196] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/b9100225d1ce44aeac332d8cf87a1055
I20260620 20:18:40.078398 11039 ts_tablet_manager.cc:1939] T b9100225d1ce44aeac332d8cf87a1055 P 5c8e0053952044f287ff0391e3155661: Deleting consensus metadata
I20260620 20:18:40.078390 11040 ts_tablet_manager.cc:1918] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:40.078467 11040 log.cc:1196] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/b9100225d1ce44aeac332d8cf87a1055
I20260620 20:18:40.078742 11040 ts_tablet_manager.cc:1939] T b9100225d1ce44aeac332d8cf87a1055 P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting consensus metadata
I20260620 20:18:40.078852 10781 tablet_service.cc:1467] Processing CreateTablet for tablet cdd1c155bc34453189f5f97bd99b1efe (DEFAULT_TABLE table=test-table [id=1d876f7ad8dc446cbeae96bd9a18a7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.079067 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd1c155bc34453189f5f97bd99b1efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.080746 11016 tablet_bootstrap.cc:492] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Bootstrap starting.
I20260620 20:18:40.081260 10488 catalog_manager.cc:4841] TS 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939): tablet b9100225d1ce44aeac332d8cf87a1055 (table test-table [id=a98f1bfd770e4d40858f90b819c68b6c]) successfully deleted
I20260620 20:18:40.081398 10488 catalog_manager.cc:4841] TS 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155): tablet b9100225d1ce44aeac332d8cf87a1055 (table test-table [id=a98f1bfd770e4d40858f90b819c68b6c]) successfully deleted
I20260620 20:18:40.081459 10641 tablet_service.cc:1467] Processing CreateTablet for tablet cdd1c155bc34453189f5f97bd99b1efe (DEFAULT_TABLE table=test-table [id=1d876f7ad8dc446cbeae96bd9a18a7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.081631 11016 tablet_bootstrap.cc:654] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.081671 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd1c155bc34453189f5f97bd99b1efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.082397 10910 tablet_service.cc:1467] Processing CreateTablet for tablet cdd1c155bc34453189f5f97bd99b1efe (DEFAULT_TABLE table=test-table [id=1d876f7ad8dc446cbeae96bd9a18a7be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.082614 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdd1c155bc34453189f5f97bd99b1efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.083535 11015 tablet_bootstrap.cc:492] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Bootstrap starting.
I20260620 20:18:40.084263 11019 tablet_bootstrap.cc:492] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Bootstrap starting.
I20260620 20:18:40.084348 11015 tablet_bootstrap.cc:654] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.084782 11016 tablet_bootstrap.cc:492] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: No bootstrap required, opened a new log
I20260620 20:18:40.084858 11016 ts_tablet_manager.cc:1397] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 20:18:40.085052 11019 tablet_bootstrap.cc:654] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.085281 11016 raft_consensus.cc:348] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.085381 11016 raft_consensus.cc:374] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.085417 11016 raft_consensus.cc:729] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.085536 11016 consensus_queue.cc:260] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.085777 11016 ts_tablet_manager.cc:1428] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:18:40.086400 11038 ts_tablet_manager.cc:1905] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.088783 11038 ts_tablet_manager.cc:1918] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:40.088871 11038 log.cc:1196] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/b9100225d1ce44aeac332d8cf87a1055
I20260620 20:18:40.089141 11038 ts_tablet_manager.cc:1939] T b9100225d1ce44aeac332d8cf87a1055 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting consensus metadata
I20260620 20:18:40.089866 11015 tablet_bootstrap.cc:492] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: No bootstrap required, opened a new log
I20260620 20:18:40.089872 11019 tablet_bootstrap.cc:492] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: No bootstrap required, opened a new log
I20260620 20:18:40.089936 11019 ts_tablet_manager.cc:1397] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260620 20:18:40.089934 11015 ts_tablet_manager.cc:1397] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260620 20:18:40.090124 10489 catalog_manager.cc:4841] TS 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539): tablet b9100225d1ce44aeac332d8cf87a1055 (table test-table [id=a98f1bfd770e4d40858f90b819c68b6c]) successfully deleted
I20260620 20:18:40.090296 11019 raft_consensus.cc:348] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.090375 11019 raft_consensus.cc:374] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.090401 11019 raft_consensus.cc:729] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8e0053952044f287ff0391e3155661, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.090497 11019 consensus_queue.cc:260] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.090531 11015 raft_consensus.cc:348] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.090636 11015 raft_consensus.cc:374] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.090667 11015 raft_consensus.cc:729] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.090692 11019 ts_tablet_manager.cc:1428] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:18:40.090775 11015 consensus_queue.cc:260] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.091005 11015 ts_tablet_manager.cc:1428] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260620 20:18:40.104250 10848 tablet.cc:2369] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:40.132258 11021 raft_consensus.cc:482] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:40.132369 11021 raft_consensus.cc:504] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.133390 11021 leader_election.cc:288] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.137460 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.133877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:43155 (local address 127.2.235.194:44093)
0620 20:18:40.134119 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.134147 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.134232 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.134475 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.134480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.134500 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:40.134618 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.134628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.135391 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.135395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.136228 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.136240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.136990 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.136994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.137092 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.137117 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.137194 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.137289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":45,"thread_start_us":43,"threads_started":1}
I20260620 20:18:40.137475 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.133931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:44093 (local address 127.2.235.195:43155)
0620 20:18:40.134028 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.134032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.134216 (+ 184us) server_negotiation.cc:407] Connection header received
0620 20:18:40.134341 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.134344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.134383 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.134455 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:40.134711 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.135282 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.136368 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.136871 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.136989 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.137037 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.137228 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.137347 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:18:40.137965 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.133568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.2.235.194:34479)
0620 20:18:40.133785 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.133799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.133918 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.134273 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.134278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.134297 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:40.134411 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.134416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.135404 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.135407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.136552 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.136562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.137480 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.137485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.137707 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.137731 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.137796 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.137859 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":58,"threads_started":1}
I20260620 20:18:40.138000 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.133631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:34479 (local address 127.2.235.193:43939)
0620 20:18:40.133764 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.133768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.133853 (+ 85us) server_negotiation.cc:407] Connection header received
0620 20:18:40.134034 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.134037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.134078 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.134121 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:40.134550 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.135280 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.136674 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.137246 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.137389 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.137428 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.137827 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.137896 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:18:40.138005 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd1c155bc34453189f5f97bd99b1efe" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661" is_pre_election: true
I20260620 20:18:40.138204 10939 raft_consensus.cc:2454] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 0.
I20260620 20:18:40.138525 10732 leader_election.cc:302] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e0053952044f287ff0391e3155661, 6d77eae8ddfe4c2ebf6294e1f4b389bc; no voters:
I20260620 20:18:40.138588 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd1c155bc34453189f5f97bd99b1efe" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" is_pre_election: true
I20260620 20:18:40.138607 11022 raft_consensus.cc:482] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:40.138697 11022 raft_consensus.cc:504] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.138784 11021 raft_consensus.cc:2781] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:40.138826 11021 raft_consensus.cc:482] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:40.138857 11021 raft_consensus.cc:3037] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.138972 10661 raft_consensus.cc:2454] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 0.
I20260620 20:18:40.139549 11022 leader_election.cc:288] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.139868 11021 raft_consensus.cc:504] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.140146 11021 leader_election.cc:288] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 election: Requested vote from peers 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.140606 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd1c155bc34453189f5f97bd99b1efe" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661"
I20260620 20:18:40.140710 10939 raft_consensus.cc:3037] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.140759 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd1c155bc34453189f5f97bd99b1efe" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccd43d49d674268aa3ab147cd37d9cc"
I20260620 20:18:40.140846 10661 raft_consensus.cc:3037] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.141654 10661 raft_consensus.cc:2454] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 1.
I20260620 20:18:40.141673 10939 raft_consensus.cc:2454] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 1.
I20260620 20:18:40.141918 10732 leader_election.cc:302] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e0053952044f287ff0391e3155661, 6d77eae8ddfe4c2ebf6294e1f4b389bc; no voters:
I20260620 20:18:40.142303 11021 raft_consensus.cc:2781] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:40.142432 11021 raft_consensus.cc:686] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 LEADER]: Becoming Leader. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Running, Role: LEADER
I20260620 20:18:40.142625 11021 consensus_queue.cc:237] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.144383 10503 catalog_manager.cc:5495] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194). New cstate: current_term: 1 leader_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:40.146049 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.139948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:42535 (local address 127.2.235.194:35539)
0620 20:18:40.140045 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.140051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.142052 (+ 2001us) server_negotiation.cc:407] Connection header received
0620 20:18:40.142197 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.142199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.142240 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.142310 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:40.142712 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.143306 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.144961 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.145478 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.145542 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.145578 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.145884 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.145945 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:18:40.146538 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.139668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:35539 (local address 127.2.235.193:42535)
0620 20:18:40.141964 (+ 2296us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.141974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.142110 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.142438 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.142441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.142453 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:40.142601 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.142609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.143480 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.143484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.144846 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.144855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.145667 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.145670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.145753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.145770 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.146298 (+ 528us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.146381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2241,"thread_start_us":88,"threads_started":1}
I20260620 20:18:40.146646 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.140070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:54771 (local address 127.2.235.195:43155)
0620 20:18:40.140182 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.140187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.142388 (+ 2201us) server_negotiation.cc:407] Connection header received
0620 20:18:40.142534 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.142537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.142576 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.142615 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:18:40.143597 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.144208 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.145446 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.145954 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.146024 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.146060 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.146464 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.146521 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:18:40.147010 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.139803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:43155 (local address 127.2.235.193:54771)
0620 20:18:40.142322 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.142333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.142440 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.142722 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.142725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.142738 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:18:40.142846 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.142851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.144330 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.144333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.145336 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.145345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.146172 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.146177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.146286 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.146311 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.146805 (+ 494us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.146878 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2458,"thread_start_us":85,"threads_started":1}
I20260620 20:18:40.147035 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd1c155bc34453189f5f97bd99b1efe" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" is_pre_election: true
I20260620 20:18:40.147390 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdd1c155bc34453189f5f97bd99b1efe" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661" is_pre_election: true
I20260620 20:18:40.147524 10939 raft_consensus.cc:2379] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ccd43d49d674268aa3ab147cd37d9cc in current term 1: Already voted for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in this term.
I20260620 20:18:40.147946 10592 leader_election.cc:302] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1ccd43d49d674268aa3ab147cd37d9cc; no voters: 5c8e0053952044f287ff0391e3155661, 6d77eae8ddfe4c2ebf6294e1f4b389bc
I20260620 20:18:40.148255 11022 raft_consensus.cc:2726] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986719, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.254@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:40.199013 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.186605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33685 (local address 127.0.0.1:33168)
0620 20:18:40.186882 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.186897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.187001 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.187235 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.187238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.187700 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:40.187799 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.187805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.188449 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.188452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.189240 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.189250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.189648 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.189652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.189727 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.194188 (+ 4461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:40.194202 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:40.195710 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:40.197585 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:40.197593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:40.197597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:40.197858 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:40.198122 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:40.198124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:40.198126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:40.198198 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:40.198564 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:40.198568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:40.198656 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.198757 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.198859 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260620 20:18:40.199111 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.186680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.2.235.254:33685)
0620 20:18:40.186827 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.186832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.186970 (+ 138us) server_negotiation.cc:407] Connection header received
0620 20:18:40.187080 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.187082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.187118 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.187180 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:40.187903 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.188347 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.189372 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.189543 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.189610 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.195855 (+ 6245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:40.195859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:40.195862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:40.195886 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:40.197470 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:40.197951 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:40.197955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:40.197957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:40.198036 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:40.198285 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:40.198288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:40.198290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:40.198473 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:40.198561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.198824 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.198942 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:18:40.202605 10503 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33168:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:18:40.203248 10503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:40.207631 10641 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbe33832d714b7199a047376b9ea6a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca67d35a8cb7423c8d1c1a9e3cc05121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:40.207815 10781 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbe33832d714b7199a047376b9ea6a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca67d35a8cb7423c8d1c1a9e3cc05121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:40.207980 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbe33832d714b7199a047376b9ea6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.208097 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbe33832d714b7199a047376b9ea6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.210767 10910 tablet_service.cc:1467] Processing CreateTablet for tablet 5cbe33832d714b7199a047376b9ea6a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca67d35a8cb7423c8d1c1a9e3cc05121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:18:40.211057 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cbe33832d714b7199a047376b9ea6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.213075 11019 tablet_bootstrap.cc:492] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: Bootstrap starting.
I20260620 20:18:40.213130 11016 tablet_bootstrap.cc:492] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Bootstrap starting.
I20260620 20:18:40.213860 11016 tablet_bootstrap.cc:654] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.213907 11019 tablet_bootstrap.cc:654] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.214512 11015 tablet_bootstrap.cc:492] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: Bootstrap starting.
I20260620 20:18:40.215279 11019 tablet_bootstrap.cc:492] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: No bootstrap required, opened a new log
I20260620 20:18:40.215342 11015 tablet_bootstrap.cc:654] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.215364 11019 ts_tablet_manager.cc:1397] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:40.215725 11019 raft_consensus.cc:348] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.215835 11019 raft_consensus.cc:374] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.215871 11019 raft_consensus.cc:729] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8e0053952044f287ff0391e3155661, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.215978 11019 consensus_queue.cc:260] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.216106 11016 tablet_bootstrap.cc:492] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: No bootstrap required, opened a new log
I20260620 20:18:40.216193 11016 ts_tablet_manager.cc:1397] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 20:18:40.216372 11019 ts_tablet_manager.cc:1428] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:40.216612 11023 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.216737 11023 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.216656 11016 raft_consensus.cc:348] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.216781 11016 raft_consensus.cc:374] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.216817 11016 raft_consensus.cc:729] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.216948 11016 consensus_queue.cc:260] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.217238 11016 ts_tablet_manager.cc:1428] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:40.217481 11048 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.217598 11048 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.218406 11015 tablet_bootstrap.cc:492] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: No bootstrap required, opened a new log
I20260620 20:18:40.218482 11015 ts_tablet_manager.cc:1397] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 20:18:40.218916 11015 raft_consensus.cc:348] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.219038 11015 raft_consensus.cc:374] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.219071 11015 raft_consensus.cc:729] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.219180 11015 consensus_queue.cc:260] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.219422 11015 ts_tablet_manager.cc:1428] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 20:18:40.219410 11022 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.219529 11022 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.222900 11022 raft_consensus.cc:482] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:40.222968 11022 raft_consensus.cc:504] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.223217 11022 leader_election.cc:288] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.223485 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbe33832d714b7199a047376b9ea6a8" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661" is_pre_election: true
I20260620 20:18:40.223498 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbe33832d714b7199a047376b9ea6a8" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" is_pre_election: true
I20260620 20:18:40.223574 10939 raft_consensus.cc:2454] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 0.
I20260620 20:18:40.223608 10801 raft_consensus.cc:2454] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 0.
I20260620 20:18:40.223832 10592 leader_election.cc:302] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccd43d49d674268aa3ab147cd37d9cc, 5c8e0053952044f287ff0391e3155661; no voters:
I20260620 20:18:40.223999 11022 raft_consensus.cc:2781] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:40.224045 11022 raft_consensus.cc:482] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:40.224102 11022 raft_consensus.cc:3037] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.224855 11022 raft_consensus.cc:504] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.225183 11022 leader_election.cc:288] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 election: Requested vote from peers 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.225468 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbe33832d714b7199a047376b9ea6a8" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc"
I20260620 20:18:40.225505 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cbe33832d714b7199a047376b9ea6a8" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661"
I20260620 20:18:40.225567 10801 raft_consensus.cc:3037] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.225584 10939 raft_consensus.cc:3037] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.226504 10801 raft_consensus.cc:2454] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 1.
I20260620 20:18:40.226504 10939 raft_consensus.cc:2454] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 1.
I20260620 20:18:40.226789 10593 leader_election.cc:302] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccd43d49d674268aa3ab147cd37d9cc, 6d77eae8ddfe4c2ebf6294e1f4b389bc; no voters:
I20260620 20:18:40.226953 11022 raft_consensus.cc:2781] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:40.227137 11022 raft_consensus.cc:686] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 LEADER]: Becoming Leader. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Running, Role: LEADER
I20260620 20:18:40.227358 11022 consensus_queue.cc:237] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.228402 11059 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: TxnStatusTablet state changed. Reason: New leader 1ccd43d49d674268aa3ab147cd37d9cc. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.228503 11059 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:40.228744 11061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:18:40.228878 11061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:18:40.229238 10504 catalog_manager.cc:5495] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193). New cstate: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:40.239820 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.236538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.0.0.1:60566)
0620 20:18:40.236656 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.236673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.236770 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.237103 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.237106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.237113 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:40.237214 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.237218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.237934 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.237940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.238486 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.238493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.238911 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.238914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.238996 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.239031 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.239569 (+ 538us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.239583 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.239647 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.239720 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 20:18:40.239816 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.236733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.2.235.193:43939)
0620 20:18:40.236849 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.236853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.236867 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:40.236934 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.236936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.236995 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.237090 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.237347 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.237837 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.238594 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.238810 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.238903 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.239156 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.239482 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.239589 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.239656 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.239700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:40.241976 10801 raft_consensus.cc:1261] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Refusing update from remote peer 1ccd43d49d674268aa3ab147cd37d9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.242029 10939 raft_consensus.cc:1261] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Refusing update from remote peer 1ccd43d49d674268aa3ab147cd37d9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.242354 11022 consensus_queue.cc:1035] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.242482 11059 consensus_queue.cc:1035] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.244000 11023 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: TxnStatusTablet state changed. Reason: New leader 1ccd43d49d674268aa3ab147cd37d9cc. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.244089 11023 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.244206 11048 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: TxnStatusTablet state changed. Reason: New leader 1ccd43d49d674268aa3ab147cd37d9cc. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.244292 11048 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.245657 11022 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.245749 11022 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:40.245998 11030 mvcc.cc:205] Tried to move back new op lower bound from 7299017606107299840 to 7299017606053560320. Current Snapshot: MvccSnapshot[applied={T|T < 7299017606107299840}]
I20260620 20:18:40.246611 11022 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.246716 11022 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 1ccd43d49d674268aa3ab147cd37d9cc: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:18:40.247515 11023 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.247605 11023 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 5c8e0053952044f287ff0391e3155661: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.247792 11048 tablet_replica.cc:439] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } } }
I20260620 20:18:40.247874 11048 tablet_replica.cc:442] T 5cbe33832d714b7199a047376b9ea6a8 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:18:40.254177 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.250978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.2.235.194:35539)
0620 20:18:40.251082 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.251086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.251098 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:40.251216 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.251218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.251266 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.251350 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.251519 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.252154 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.252854 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.253159 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.253262 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.253516 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.253901 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.253993 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.254039 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.254079 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:18:40.263034 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.259888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:60461 (local address 127.2.235.193:43939)
0620 20:18:40.259995 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.259999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.260143 (+ 144us) server_negotiation.cc:407] Connection header received
0620 20:18:40.260228 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.260231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.260270 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.260337 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.260526 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.261189 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.262109 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.262414 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.262499 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.262702 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.262771 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.262833 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.262896 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.262935 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:18:40.263304 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.259786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.2.235.194:60461)
0620 20:18:40.260073 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.260084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.260155 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.260350 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.260351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.260356 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:40.260428 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.260431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.261301 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.261303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.262012 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.262020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.262511 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.262514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.262571 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.262602 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.262841 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.262845 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.262887 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.262936 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260620 20:18:40.269033 10939 raft_consensus.cc:1261] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Refusing update from remote peer 6d77eae8ddfe4c2ebf6294e1f4b389bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.269033 10661 raft_consensus.cc:1261] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Refusing update from remote peer 6d77eae8ddfe4c2ebf6294e1f4b389bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.269426 11021 consensus_queue.cc:1035] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.270200 11021 consensus_queue.cc:1035] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.274464 11028 mvcc.cc:205] Tried to move back new op lower bound from 7299017606218301440 to 7299017605706199040. Current Snapshot: MvccSnapshot[applied={T|T < 7299017606218301440}]
I20260620 20:18:40.293162 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.290045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:49021 (local address 127.2.235.194:35539)
0620 20:18:40.290152 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.290156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.290331 (+ 175us) server_negotiation.cc:407] Connection header received
0620 20:18:40.290430 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.290432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.290464 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.290525 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.290709 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.291207 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.292325 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.292504 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.292570 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.292848 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.292896 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.292948 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.293062 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.293093 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:18:40.293327 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.289984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:35539 (local address 127.2.235.193:49021)
0620 20:18:40.290255 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.290267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.290351 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.290547 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.290550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.290555 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:40.290646 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.290651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.291316 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.291319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.292231 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.292240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.292615 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.292618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.292699 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.292747 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.292975 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.292979 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.293055 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.293116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":108,"threads_started":1}
I20260620 20:18:40.316272 10504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.316404 10504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.318378 10504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Sending DeleteTablet for 3 replicas of tablet cdd1c155bc34453189f5f97bd99b1efe
I20260620 20:18:40.318912 10910 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd1c155bc34453189f5f97bd99b1efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:57298
I20260620 20:18:40.318938 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd1c155bc34453189f5f97bd99b1efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 20:18:40.319077 11039 tablet_replica.cc:330] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: stopping tablet replica
I20260620 20:18:40.319082 11038 tablet_replica.cc:330] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: stopping tablet replica
I20260620 20:18:40.319170 11038 raft_consensus.cc:2229] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:40.319175 11039 raft_consensus.cc:2229] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.319275 11039 raft_consensus.cc:2258] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.319378 11038 raft_consensus.cc:2258] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.319773 10641 tablet_service.cc:1514] Processing DeleteTablet for tablet cdd1c155bc34453189f5f97bd99b1efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:60558
I20260620 20:18:40.319917 11040 tablet_replica.cc:330] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: stopping tablet replica
I20260620 20:18:40.320013 11040 raft_consensus.cc:2229] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.320096 11040 raft_consensus.cc:2258] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.320091 11038 ts_tablet_manager.cc:1905] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.320266 11039 ts_tablet_manager.cc:1905] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.320729 11040 ts_tablet_manager.cc:1905] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.322829 11038 ts_tablet_manager.cc:1918] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:40.322832 11039 ts_tablet_manager.cc:1918] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:40.322889 11038 log.cc:1196] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/cdd1c155bc34453189f5f97bd99b1efe
I20260620 20:18:40.322892 11039 log.cc:1196] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/cdd1c155bc34453189f5f97bd99b1efe
I20260620 20:18:40.323127 11039 ts_tablet_manager.cc:1939] T cdd1c155bc34453189f5f97bd99b1efe P 5c8e0053952044f287ff0391e3155661: Deleting consensus metadata
I20260620 20:18:40.323139 11038 ts_tablet_manager.cc:1939] T cdd1c155bc34453189f5f97bd99b1efe P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting consensus metadata
I20260620 20:18:40.323323 11040 ts_tablet_manager.cc:1918] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:40.323377 11040 log.cc:1196] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/cdd1c155bc34453189f5f97bd99b1efe
I20260620 20:18:40.323601 11040 ts_tablet_manager.cc:1939] T cdd1c155bc34453189f5f97bd99b1efe P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting consensus metadata
I20260620 20:18:40.324792 10489 catalog_manager.cc:4841] TS 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539): tablet cdd1c155bc34453189f5f97bd99b1efe (table test-table [id=1d876f7ad8dc446cbeae96bd9a18a7be]) successfully deleted
I20260620 20:18:40.327498 10488 catalog_manager.cc:4841] TS 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939): tablet cdd1c155bc34453189f5f97bd99b1efe (table test-table [id=1d876f7ad8dc446cbeae96bd9a18a7be]) successfully deleted
I20260620 20:18:40.327634 10488 catalog_manager.cc:4841] TS 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155): tablet cdd1c155bc34453189f5f97bd99b1efe (table test-table [id=1d876f7ad8dc446cbeae96bd9a18a7be]) successfully deleted
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:18:40.353775 10553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:40.346609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.2.235.254:33685)
0620 20:18:40.346720 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.346724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.346972 (+ 248us) server_negotiation.cc:407] Connection header received
0620 20:18:40.347089 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.347092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.347144 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.347230 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:40.348001 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.348862 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.350008 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.350321 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.350419 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.353666 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33176: BlockingRecv error: recv got EOF from 127.0.0.1:33176 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM
I20260620 20:18:40.364058 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.355178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.2.235.254:33685)
0620 20:18:40.355263 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.355267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.355612 (+ 345us) server_negotiation.cc:407] Connection header received
0620 20:18:40.355687 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.355690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.355733 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.355804 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:40.356426 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.357212 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.358211 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.358474 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.358567 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.360882 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:40.360885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:40.360887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:40.360908 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:40.362459 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:40.362907 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:40.362910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:40.362912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:40.362982 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:40.363251 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:40.363254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:40.363256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:40.363429 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:40.363554 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.363818 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.363924 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:18:40.366659 10504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:40.367115 10504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:40.371686 10641 tablet_service.cc:1467] Processing CreateTablet for tablet ee08b30e98304985ada0dc3fd8d57245 (DEFAULT_TABLE table=test-table [id=44431f0bf78d480bb4ebe8d0c8419519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.371860 10781 tablet_service.cc:1467] Processing CreateTablet for tablet ee08b30e98304985ada0dc3fd8d57245 (DEFAULT_TABLE table=test-table [id=44431f0bf78d480bb4ebe8d0c8419519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.371974 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08b30e98304985ada0dc3fd8d57245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.372109 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08b30e98304985ada0dc3fd8d57245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.373950 11015 tablet_bootstrap.cc:492] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Bootstrap starting.
I20260620 20:18:40.374177 10910 tablet_service.cc:1467] Processing CreateTablet for tablet ee08b30e98304985ada0dc3fd8d57245 (DEFAULT_TABLE table=test-table [id=44431f0bf78d480bb4ebe8d0c8419519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.374410 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08b30e98304985ada0dc3fd8d57245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.374796 11015 tablet_bootstrap.cc:654] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.376034 11015 tablet_bootstrap.cc:492] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: No bootstrap required, opened a new log
I20260620 20:18:40.376111 11015 ts_tablet_manager.cc:1397] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:18:40.376763 11019 tablet_bootstrap.cc:492] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Bootstrap starting.
I20260620 20:18:40.376850 11016 tablet_bootstrap.cc:492] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Bootstrap starting.
I20260620 20:18:40.376950 11015 raft_consensus.cc:348] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.377056 11015 raft_consensus.cc:374] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.377089 11015 raft_consensus.cc:729] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.377210 11015 consensus_queue.cc:260] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.377601 11019 tablet_bootstrap.cc:654] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.377761 11016 tablet_bootstrap.cc:654] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.378722 11015 ts_tablet_manager.cc:1428] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:18:40.378925 11016 tablet_bootstrap.cc:492] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: No bootstrap required, opened a new log
I20260620 20:18:40.379019 11016 ts_tablet_manager.cc:1397] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 20:18:40.379432 11016 raft_consensus.cc:348] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.379537 11016 raft_consensus.cc:374] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.379570 11016 raft_consensus.cc:729] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.379684 11016 consensus_queue.cc:260] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.379940 11016 ts_tablet_manager.cc:1428] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:40.380025 11019 tablet_bootstrap.cc:492] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: No bootstrap required, opened a new log
I20260620 20:18:40.380097 11019 ts_tablet_manager.cc:1397] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:18:40.380390 11019 raft_consensus.cc:348] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.380472 11019 raft_consensus.cc:374] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.380494 11019 raft_consensus.cc:729] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8e0053952044f287ff0391e3155661, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.380578 11019 consensus_queue.cc:260] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.380776 11019 ts_tablet_manager.cc:1428] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:40.392622 11048 raft_consensus.cc:482] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:40.392706 11048 raft_consensus.cc:504] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.393009 11048 leader_election.cc:288] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.393334 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08b30e98304985ada0dc3fd8d57245" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661" is_pre_election: true
I20260620 20:18:40.393443 10939 raft_consensus.cc:2454] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 0.
I20260620 20:18:40.393527 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08b30e98304985ada0dc3fd8d57245" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" is_pre_election: true
I20260620 20:18:40.393625 10661 raft_consensus.cc:2454] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 0.
I20260620 20:18:40.393678 10732 leader_election.cc:302] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e0053952044f287ff0391e3155661, 6d77eae8ddfe4c2ebf6294e1f4b389bc; no voters:
I20260620 20:18:40.394146 11048 raft_consensus.cc:2781] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:40.394201 11048 raft_consensus.cc:482] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:40.394248 11048 raft_consensus.cc:3037] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.395164 11048 raft_consensus.cc:504] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.395454 11048 leader_election.cc:288] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 election: Requested vote from peers 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.395795 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08b30e98304985ada0dc3fd8d57245" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661"
I20260620 20:18:40.395889 10939 raft_consensus.cc:3037] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.396847 10939 raft_consensus.cc:2454] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 1.
I20260620 20:18:40.397272 10661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08b30e98304985ada0dc3fd8d57245" candidate_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ccd43d49d674268aa3ab147cd37d9cc"
I20260620 20:18:40.397392 10661 raft_consensus.cc:3037] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.398288 10661 raft_consensus.cc:2454] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d77eae8ddfe4c2ebf6294e1f4b389bc in term 1.
I20260620 20:18:40.398478 10732 leader_election.cc:302] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c8e0053952044f287ff0391e3155661, 6d77eae8ddfe4c2ebf6294e1f4b389bc; no voters:
I20260620 20:18:40.398798 11048 raft_consensus.cc:2781] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:40.398878 11048 raft_consensus.cc:686] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 LEADER]: Becoming Leader. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Running, Role: LEADER
I20260620 20:18:40.398984 11048 consensus_queue.cc:237] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.400348 10504 catalog_manager.cc:5495] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194). New cstate: current_term: 1 leader_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:40.412647 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.409417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.2.235.194:35539)
0620 20:18:40.409507 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.409511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.409524 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:40.409559 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.409561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.409618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.409688 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.409905 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.410602 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.411651 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.411953 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.412041 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.412293 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.412359 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.412420 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.412503 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.412546 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 20:18:40.414363 10661 raft_consensus.cc:1261] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Refusing update from remote peer 6d77eae8ddfe4c2ebf6294e1f4b389bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.414381 10939 raft_consensus.cc:1261] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Refusing update from remote peer 6d77eae8ddfe4c2ebf6294e1f4b389bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.414732 11048 consensus_queue.cc:1035] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.414860 11072 consensus_queue.cc:1035] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.421132 10503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.421269 10503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.423245 10503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Sending DeleteTablet for 3 replicas of tablet ee08b30e98304985ada0dc3fd8d57245
I20260620 20:18:40.423704 10641 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08b30e98304985ada0dc3fd8d57245 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:60558
I20260620 20:18:40.423739 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08b30e98304985ada0dc3fd8d57245 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 20:18:40.423874 11040 tablet_replica.cc:330] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: stopping tablet replica
I20260620 20:18:40.423874 11038 tablet_replica.cc:330] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: stopping tablet replica
I20260620 20:18:40.423983 11040 raft_consensus.cc:2229] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.423987 11038 raft_consensus.cc:2229] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:18:40.424089 11040 raft_consensus.cc:2258] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.424144 11038 raft_consensus.cc:2258] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.424552 10910 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08b30e98304985ada0dc3fd8d57245 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:57298
I20260620 20:18:40.424686 11039 tablet_replica.cc:330] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: stopping tablet replica
I20260620 20:18:40.424737 11038 ts_tablet_manager.cc:1905] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.424763 11039 raft_consensus.cc:2229] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.424870 11039 raft_consensus.cc:2258] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.425237 10504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:18:40.425587 10504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:18:40.425588 11040 ts_tablet_manager.cc:1905] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.427121 11038 ts_tablet_manager.cc:1918] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:40.427199 11038 log.cc:1196] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/ee08b30e98304985ada0dc3fd8d57245
I20260620 20:18:40.427448 11038 ts_tablet_manager.cc:1939] T ee08b30e98304985ada0dc3fd8d57245 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting consensus metadata
I20260620 20:18:40.428032 10489 catalog_manager.cc:4841] TS 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539): tablet ee08b30e98304985ada0dc3fd8d57245 (table test-table [id=44431f0bf78d480bb4ebe8d0c8419519]) successfully deleted
I20260620 20:18:40.428066 11040 ts_tablet_manager.cc:1918] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:40.428124 11040 log.cc:1196] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/ee08b30e98304985ada0dc3fd8d57245
I20260620 20:18:40.428397 11040 ts_tablet_manager.cc:1939] T ee08b30e98304985ada0dc3fd8d57245 P 1ccd43d49d674268aa3ab147cd37d9cc: Deleting consensus metadata
I20260620 20:18:40.429114 10488 catalog_manager.cc:4841] TS 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939): tablet ee08b30e98304985ada0dc3fd8d57245 (table test-table [id=44431f0bf78d480bb4ebe8d0c8419519]) successfully deleted
I20260620 20:18:40.429934 10641 tablet_service.cc:1467] Processing CreateTablet for tablet e59c53790b7b4e108a7ccebee287bd12 (DEFAULT_TABLE table=test-table [id=302c57f79bca47e0a52c6cdabb4e7893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.430083 10781 tablet_service.cc:1467] Processing CreateTablet for tablet e59c53790b7b4e108a7ccebee287bd12 (DEFAULT_TABLE table=test-table [id=302c57f79bca47e0a52c6cdabb4e7893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.430152 10641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59c53790b7b4e108a7ccebee287bd12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.430272 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59c53790b7b4e108a7ccebee287bd12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.430296 10910 tablet_service.cc:1467] Processing CreateTablet for tablet e59c53790b7b4e108a7ccebee287bd12 (DEFAULT_TABLE table=test-table [id=302c57f79bca47e0a52c6cdabb4e7893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:18:40.430513 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59c53790b7b4e108a7ccebee287bd12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:40.432101 11016 tablet_bootstrap.cc:492] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Bootstrap starting.
I20260620 20:18:40.432427 11039 ts_tablet_manager.cc:1905] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.432657 11019 tablet_bootstrap.cc:492] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Bootstrap starting.
I20260620 20:18:40.432691 11015 tablet_bootstrap.cc:492] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc: Bootstrap starting.
I20260620 20:18:40.432844 11016 tablet_bootstrap.cc:654] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.433342 11019 tablet_bootstrap.cc:654] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.433887 11015 tablet_bootstrap.cc:654] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:40.434396 11039 ts_tablet_manager.cc:1918] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:18:40.434473 11039 log.cc:1196] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/ee08b30e98304985ada0dc3fd8d57245
I20260620 20:18:40.434741 11039 ts_tablet_manager.cc:1939] T ee08b30e98304985ada0dc3fd8d57245 P 5c8e0053952044f287ff0391e3155661: Deleting consensus metadata
I20260620 20:18:40.434984 11015 tablet_bootstrap.cc:492] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc: No bootstrap required, opened a new log
I20260620 20:18:40.435086 11015 ts_tablet_manager.cc:1397] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:40.435457 10488 catalog_manager.cc:4841] TS 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155): tablet ee08b30e98304985ada0dc3fd8d57245 (table test-table [id=44431f0bf78d480bb4ebe8d0c8419519]) successfully deleted
I20260620 20:18:40.435501 11015 raft_consensus.cc:348] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.435596 11015 raft_consensus.cc:374] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.435655 11015 raft_consensus.cc:729] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.435781 11015 consensus_queue.cc:260] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.436043 11015 ts_tablet_manager.cc:1428] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:40.436051 11019 tablet_bootstrap.cc:492] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: No bootstrap required, opened a new log
I20260620 20:18:40.436106 11019 ts_tablet_manager.cc:1397] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 20:18:40.436317 11016 tablet_bootstrap.cc:492] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: No bootstrap required, opened a new log
I20260620 20:18:40.436391 11016 ts_tablet_manager.cc:1397] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 20:18:40.436515 11019 raft_consensus.cc:348] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.436622 11019 raft_consensus.cc:374] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.436657 11019 raft_consensus.cc:729] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8e0053952044f287ff0391e3155661, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.436786 11019 consensus_queue.cc:260] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.436847 11016 raft_consensus.cc:348] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.436944 11016 raft_consensus.cc:374] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:40.436982 11016 raft_consensus.cc:729] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d77eae8ddfe4c2ebf6294e1f4b389bc, State: Initialized, Role: FOLLOWER
I20260620 20:18:40.437067 11019 ts_tablet_manager.cc:1428] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:18:40.437091 11016 consensus_queue.cc:260] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.437415 11016 ts_tablet_manager.cc:1428] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:18:40.559774 11066 raft_consensus.cc:482] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:18:40.559887 11066 raft_consensus.cc:504] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.560400 11066 leader_election.cc:288] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.560781 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c53790b7b4e108a7ccebee287bd12" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" is_pre_election: true
I20260620 20:18:40.560835 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c53790b7b4e108a7ccebee287bd12" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661" is_pre_election: true
I20260620 20:18:40.560940 10801 raft_consensus.cc:2454] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 0.
I20260620 20:18:40.560961 10939 raft_consensus.cc:2454] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 0.
I20260620 20:18:40.561223 10592 leader_election.cc:302] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccd43d49d674268aa3ab147cd37d9cc, 5c8e0053952044f287ff0391e3155661; no voters:
I20260620 20:18:40.561415 11066 raft_consensus.cc:2781] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:18:40.561492 11066 raft_consensus.cc:482] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:18:40.561539 11066 raft_consensus.cc:3037] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.562664 11066 raft_consensus.cc:504] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.563006 11066 leader_election.cc:288] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 election: Requested vote from peers 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539), 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155)
I20260620 20:18:40.563275 10801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c53790b7b4e108a7ccebee287bd12" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc"
I20260620 20:18:40.563386 10801 raft_consensus.cc:3037] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.563375 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59c53790b7b4e108a7ccebee287bd12" candidate_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c8e0053952044f287ff0391e3155661"
I20260620 20:18:40.563484 10939 raft_consensus.cc:3037] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:40.564471 10801 raft_consensus.cc:2454] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 1.
I20260620 20:18:40.564495 10939 raft_consensus.cc:2454] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccd43d49d674268aa3ab147cd37d9cc in term 1.
I20260620 20:18:40.564741 10592 leader_election.cc:302] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccd43d49d674268aa3ab147cd37d9cc, 5c8e0053952044f287ff0391e3155661; no voters:
I20260620 20:18:40.564901 11066 raft_consensus.cc:2781] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:40.565001 11066 raft_consensus.cc:686] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [term 1 LEADER]: Becoming Leader. State: Replica: 1ccd43d49d674268aa3ab147cd37d9cc, State: Running, Role: LEADER
I20260620 20:18:40.565126 11066 consensus_queue.cc:237] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } }
I20260620 20:18:40.566533 10503 catalog_manager.cc:5495] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193). New cstate: current_term: 1 leader_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ccd43d49d674268aa3ab147cd37d9cc" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 43939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 } health_report { overall_health: UNKNOWN } } }
I20260620 20:18:40.594362 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.591432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.2.235.193:43939)
0620 20:18:40.591538 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.591543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.591574 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:18:40.591719 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.591722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.591767 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.591847 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.592085 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.592550 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.593451 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.593646 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.593724 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.593974 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.594034 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.594082 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.594157 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.594200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:18:40.601997 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.599140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:41151 (local address 127.2.235.193:43939)
0620 20:18:40.599240 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:40.599245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:40.599288 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:18:40.599405 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:40.599408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:40.599439 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:40.599503 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:40.599728 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.600170 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.601066 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.601285 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.601394 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.601578 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.601630 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.601689 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:40.601825 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 20:18:40.601873 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
I20260620 20:18:40.602032 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.599092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.2.235.193:41151)
0620 20:18:40.599219 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.599232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:40.599329 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:40.599520 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:40.599522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:40.599527 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:18:40.599618 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.599622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.600315 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.600318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.600961 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:40.600975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:40.601410 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:40.601412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:40.601469 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:40.601491 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:18:40.601711 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:18:40.601732 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:18:40.601796 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:18:40.601865 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":16}
I20260620 20:18:40.607018 10801 raft_consensus.cc:1261] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Refusing update from remote peer 1ccd43d49d674268aa3ab147cd37d9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.607436 11066 consensus_queue.cc:1035] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d77eae8ddfe4c2ebf6294e1f4b389bc" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 35539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.609764 10939 raft_consensus.cc:1261] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Refusing update from remote peer 1ccd43d49d674268aa3ab147cd37d9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:40.610364 11022 consensus_queue.cc:1035] T e59c53790b7b4e108a7ccebee287bd12 P 1ccd43d49d674268aa3ab147cd37d9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c8e0053952044f287ff0391e3155661" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:40.639900 10503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.640033 10503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:18:40.642024 10503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffa09ff7e83847a7a289394b379175a9: Sending DeleteTablet for 3 replicas of tablet e59c53790b7b4e108a7ccebee287bd12
I20260620 20:18:40.642462 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet e59c53790b7b4e108a7ccebee287bd12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260620 20:18:40.642462 10910 tablet_service.cc:1514] Processing DeleteTablet for tablet e59c53790b7b4e108a7ccebee287bd12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:18:40 UTC) from {username='oryx', principal='oryx/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:57298
I20260620 20:18:40.642596 11039 tablet_replica.cc:330] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: stopping tablet replica
I20260620 20:18:40.642596 11038 tablet_replica.cc:330] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: stopping tablet replica
I20260620 20:18:40.642678 11038 raft_consensus.cc:2229] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.642678 11039 raft_consensus.cc:2229] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:18:40.642760 11038 raft_consensus.cc:2258] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.642791 11039 raft_consensus.cc:2258] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:18:40.643028 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 10576
I20260620 20:18:40.643386 11039 ts_tablet_manager.cc:1905] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.643436 11038 ts_tablet_manager.cc:1905] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:18:40.645622 11039 ts_tablet_manager.cc:1918] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:40.645622 11038 ts_tablet_manager.cc:1918] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:18:40.645692 11038 log.cc:1196] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/e59c53790b7b4e108a7ccebee287bd12
I20260620 20:18:40.645692 11039 log.cc:1196] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/e59c53790b7b4e108a7ccebee287bd12
I20260620 20:18:40.645939 11038 ts_tablet_manager.cc:1939] T e59c53790b7b4e108a7ccebee287bd12 P 6d77eae8ddfe4c2ebf6294e1f4b389bc: Deleting consensus metadata
I20260620 20:18:40.645939 11039 ts_tablet_manager.cc:1939] T e59c53790b7b4e108a7ccebee287bd12 P 5c8e0053952044f287ff0391e3155661: Deleting consensus metadata
I20260620 20:18:40.646503 10488 catalog_manager.cc:4841] TS 5c8e0053952044f287ff0391e3155661 (127.2.235.195:43155): tablet e59c53790b7b4e108a7ccebee287bd12 (table test-table [id=302c57f79bca47e0a52c6cdabb4e7893]) successfully deleted
I20260620 20:18:40.646605 10489 catalog_manager.cc:4841] TS 6d77eae8ddfe4c2ebf6294e1f4b389bc (127.2.235.194:35539): tablet e59c53790b7b4e108a7ccebee287bd12 (table test-table [id=302c57f79bca47e0a52c6cdabb4e7893]) successfully deleted
W20260620 20:18:40.651193 10488 connection.cc:535] client connection to 127.2.235.193:43939 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 20:18:40.651350 10488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:18:40.651399 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 10716
I20260620 20:18:40.653609 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:40.653107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:43939 (local address 127.0.0.1:60574)
0620 20:18:40.653468 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:40.653535 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":56,"threads_started":1}
W20260620 20:18:40.653776 10488 catalog_manager.cc:4584] TS 1ccd43d49d674268aa3ab147cd37d9cc (127.2.235.193:43939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e59c53790b7b4e108a7ccebee287bd12: Network error: Client connection negotiation failed: client connection to 127.2.235.193:43939: connect: Connection refused (error 111)
I20260620 20:18:40.658066 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 10856
I20260620 20:18:40.664310 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 10472
2026-06-20T20:18:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2740 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[11108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[11108](info): set up 2 sockets
Jun 20 20:18:40 dist-test-slave-8z53 krb5kdc[11108](info): commencing operation
krb5kdc: starting...
W20260620 20:18:41.629993 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:41 dist-test-slave-8z53 krb5kdc[11108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:41.735751 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34653
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:34653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:41.835934 11121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:41.836136 11121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:41.836172 11121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:41.839869 11121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:41.839911 11121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:41.839928 11121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:41.839943 11121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:41.839993 11121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:41.844449 11121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:34653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:34653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:41.845228 11121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:41.846066 11121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:41.853363 11126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:41.853440 11121 server_base.cc:1034] running on GCE node
W20260620 20:18:41.853359 11127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:41.853355 11129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:41.854060 11121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:41.854933 11121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:41.856069 11121 hybrid_clock.cc:648] HybridClock initialized: now 1781986721856038 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T20:18:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1198 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:18:41 dist-test-slave-8z53 krb5kdc[11134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:18:41 dist-test-slave-8z53 krb5kdc[11134](info): set up 2 sockets
Jun 20 20:18:41 dist-test-slave-8z53 krb5kdc[11134](info): commencing operation
krb5kdc: starting...
W20260620 20:18:42.841737 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:18:42 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:18:42.947959 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33291
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:33291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:43.052894 11147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:43.053098 11147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:43.053162 11147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:43.056838 11147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:43.056893 11147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:43.056915 11147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:43.056959 11147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:43.056995 11147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:43.061793 11147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:33291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:43.062667 11147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:43.063541 11147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:43.071050 11153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.071064 11155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.071064 11152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.071295 11147 server_base.cc:1034] running on GCE node
I20260620 20:18:43.071844 11147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:43.072705 11147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:43.073933 11147 hybrid_clock.cc:648] HybridClock initialized: now 1781986723073904 us; error 27 us; skew 500 ppm
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:43.076253 11147 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:43.077025 11147 webserver.cc:460] Webserver started at http://127.2.235.254:33619/ using document root <none> and password file <none>
I20260620 20:18:43.077587 11147 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:43.077699 11147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:43.077898 11147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:43.079648 11147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "8b3f328d5dfa4b6287e17c6be261f466"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "92b4ecfb36f75dcb62e217e46977e15e"
server_key_iv: "fe3c97f953eda036f252c3dd5a783a6d"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.080147 11147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "8b3f328d5dfa4b6287e17c6be261f466"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "92b4ecfb36f75dcb62e217e46977e15e"
server_key_iv: "fe3c97f953eda036f252c3dd5a783a6d"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.083319 11147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:18:43.085438 11162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.086475 11147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:43.086615 11147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "8b3f328d5dfa4b6287e17c6be261f466"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "92b4ecfb36f75dcb62e217e46977e15e"
server_key_iv: "fe3c97f953eda036f252c3dd5a783a6d"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.086717 11147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:43.102442 11147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:43.104576 11147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:43.104737 11147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:43.110777 11147 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:33291
I20260620 20:18:43.110790 11214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:33291 every 8 connection(s)
I20260620 20:18:43.111522 11147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:43.112730 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11147
I20260620 20:18:43.112862 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:43.113107 2991 external_mini_cluster.cc:1428] Setting key b89ec6d11cdd77e148c83dce435dcb74
I20260620 20:18:43.114044 11215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:43.118882 11215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Bootstrap starting.
I20260620 20:18:43.120860 11215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Neither blocks nor log segments found. Creating new log.
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:43.121670 11215 log.cc:824] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:43.123466 11215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: No bootstrap required, opened a new log
I20260620 20:18:43.126045 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.114377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.2.235.254:33291)
0620 20:18:43.114766 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:43.114780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:43.114809 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:18:43.115331 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:43.115344 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:43.115598 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:43.115922 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:43.116678 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.117707 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.118550 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.119020 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.119180 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.121608 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:43.121613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:43.121623 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:43.121644 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:43.123595 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.124069 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.124073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.124077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.124179 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.124451 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.124453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.124454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.124805 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:43.124934 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:43.125205 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 20:18:43.125465 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
I20260620 20:18:43.127241 11215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:43.127460 11215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:43.127548 11215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3f328d5dfa4b6287e17c6be261f466, State: Initialized, Role: FOLLOWER
I20260620 20:18:43.127980 11215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:43.128116 11215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:43.128176 11215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:43.128270 11215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:43.129053 11215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:43.129418 11215 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b3f328d5dfa4b6287e17c6be261f466; no voters:
I20260620 20:18:43.129735 11215 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:43.129833 11220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:43.130087 11220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3f328d5dfa4b6287e17c6be261f466, State: Running, Role: LEADER
I20260620 20:18:43.130707 11220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:43.130908 11215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:43.132273 11221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b3f328d5dfa4b6287e17c6be261f466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } } }
I20260620 20:18:43.132316 11222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3f328d5dfa4b6287e17c6be261f466. Latest consensus state: current_term: 1 leader_uuid: "8b3f328d5dfa4b6287e17c6be261f466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } } }
I20260620 20:18:43.132400 11221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:43.132402 11222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:43.132754 11232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:43.135200 11232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:43.140019 11232 catalog_manager.cc:1347] Generated new cluster ID: 395f9baad97d443b8d8638ac94f3fdb2
I20260620 20:18:43.140076 11232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:43.149570 11232 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:43.150533 11232 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:43.159272 11232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Generated new TSK 0
I20260620 20:18:43.159852 11232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:18:43.189958 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33291
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:43.286230 11241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:43.286417 11241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:43.286512 11241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:43.290042 11241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:43.290086 11241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:43.290186 11241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:43.294516 11241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:43.295262 11241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:43.296008 11241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:43.303928 11249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.303953 11247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.303987 11246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.304147 11241 server_base.cc:1034] running on GCE node
I20260620 20:18:43.304581 11241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:43.305061 11241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:43.306226 11241 hybrid_clock.cc:648] HybridClock initialized: now 1781986723306202 us; error 38 us; skew 500 ppm
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:43.308780 11241 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:18:43.309567 11241 webserver.cc:460] Webserver started at http://127.2.235.193:39533/ using document root <none> and password file <none>
I20260620 20:18:43.310022 11241 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:43.310097 11241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:43.310281 11241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:43.311993 11241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "3762c7a4139b49f288097241c875e4a0"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "950f648e1426b1496332e0de309fefac"
server_key_iv: "c2460410344c0357686106d249d4330b"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.312503 11241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "3762c7a4139b49f288097241c875e4a0"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "950f648e1426b1496332e0de309fefac"
server_key_iv: "c2460410344c0357686106d249d4330b"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.315762 11241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 20:18:43.317912 11256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.319074 11241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:18:43.319212 11241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "3762c7a4139b49f288097241c875e4a0"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "950f648e1426b1496332e0de309fefac"
server_key_iv: "c2460410344c0357686106d249d4330b"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.319329 11241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:43.342752 11241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:43.345223 11241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:43.345440 11241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:43.345934 11241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:43.346844 11241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:43.346931 11241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.347004 11241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:43.347049 11241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.354915 11241 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:32973
I20260620 20:18:43.354947 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:32973 every 8 connection(s)
I20260620 20:18:43.355746 11241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:43.364745 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11241
I20260620 20:18:43.364882 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:43.365185 2991 external_mini_cluster.cc:1428] Setting key bf254ea43e0c9b634918caf41ab5c586
I20260620 20:18:43.367964 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.357389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:45221 (local address 127.2.235.254:33291)
0620 20:18:43.357541 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:43.357546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:43.358465 (+ 919us) server_negotiation.cc:407] Connection header received
0620 20:18:43.359208 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:43.359211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:43.359251 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:43.359313 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:43.360253 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.360718 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.361874 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.362096 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.362188 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.364357 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:43.364360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:43.364362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:43.364387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:43.366054 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.366538 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.366543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.366545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.366626 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.366962 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.366966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.366968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.367143 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:43.367258 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:43.367755 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 20:18:43.367859 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:18:43.369104 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.357760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:45221)
0620 20:18:43.358341 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:43.358375 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:43.359018 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:43.359446 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:43.359452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:43.359735 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:43.360124 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:43.360135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.360807 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.360809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:43.361762 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:43.361771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.362225 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.362227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:43.362346 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.362828 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:43.362842 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:43.364197 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:43.366168 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:43.366174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:43.366185 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:43.366416 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:43.366718 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:43.366722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:43.366724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:43.366858 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:43.367256 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:43.367263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:43.367478 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 20:18:43.368164 (+ 686us) client_negotiation.cc:240] Negotiation successful
0620 20:18:43.368478 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":99,"threads_started":1}
I20260620 20:18:43.370221 11371 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33291
I20260620 20:18:43.370427 11371 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:43.370930 11371 heartbeater.cc:511] Master 127.2.235.254:33291 requested a full tablet report, sending...
I20260620 20:18:43.372251 11179 ts_manager.cc:194] Registered new tserver with Master: 3762c7a4139b49f288097241c875e4a0 (127.2.235.193:32973)
I20260620 20:18:43.373332 11179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:45221
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:18:43.389170 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33291
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:43.484483 11376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:43.484704 11376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:43.484766 11376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:43.488396 11376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:43.488466 11376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:43.488538 11376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:43.492872 11376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:43.493678 11376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:43.494439 11376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:43.502812 11382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.502853 11381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.502925 11384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.502941 11376 server_base.cc:1034] running on GCE node
I20260620 20:18:43.503441 11376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:43.503911 11376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:43.505052 11376 hybrid_clock.cc:648] HybridClock initialized: now 1781986723505043 us; error 21 us; skew 500 ppm
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:43.507493 11376 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:18:43.508241 11376 webserver.cc:460] Webserver started at http://127.2.235.194:33075/ using document root <none> and password file <none>
I20260620 20:18:43.508700 11376 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:43.508754 11376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:43.508966 11376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:43.510617 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "ed7c8fba7f4245c6aeb0ee6c2a6db5cd"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "71effa6b85fc1c54468b11817b81ea11"
server_key_iv: "9b21dea509829a875e0fb84800f54b41"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.511080 11376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "ed7c8fba7f4245c6aeb0ee6c2a6db5cd"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "71effa6b85fc1c54468b11817b81ea11"
server_key_iv: "9b21dea509829a875e0fb84800f54b41"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.514045 11376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:18:43.516081 11391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.517171 11376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:43.517310 11376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "ed7c8fba7f4245c6aeb0ee6c2a6db5cd"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "71effa6b85fc1c54468b11817b81ea11"
server_key_iv: "9b21dea509829a875e0fb84800f54b41"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.517392 11376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:43.535560 11376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:43.538156 11376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:43.538275 11376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:43.538688 11376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:43.539319 11376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:43.539358 11376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.539391 11376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:43.539405 11376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.547004 11376 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:44799
I20260620 20:18:43.547029 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:44799 every 8 connection(s)
I20260620 20:18:43.547796 11376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:43.553903 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11376
I20260620 20:18:43.554014 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:18:43.554301 2991 external_mini_cluster.cc:1428] Setting key 5bc5d041afd6367e6ca13bab51abc03b
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:43.559624 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.549359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:44291 (local address 127.2.235.254:33291)
0620 20:18:43.549475 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:43.549479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:43.550294 (+ 815us) server_negotiation.cc:407] Connection header received
0620 20:18:43.550963 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:43.550965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:43.551005 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:43.551070 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:43.552083 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.552583 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.553661 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.553849 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.553919 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.556158 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:43.556163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:43.556168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:43.556197 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:43.557801 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.558325 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.558327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.558328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.558389 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.558714 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.558716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.558717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.558855 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:43.558921 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:43.559449 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 20:18:43.559524 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:18:43.560779 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.549650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:44291)
0620 20:18:43.550157 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:43.550207 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:43.550798 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:43.551211 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:43.551216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:43.551514 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:43.551958 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:43.551970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.552681 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.552683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:43.553575 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:43.553582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.553982 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.553987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:43.554162 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.554630 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:43.554647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:43.556015 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:43.557902 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:43.557911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:43.557924 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:43.558241 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:43.558471 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:43.558477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:43.558480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:43.558634 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:43.558952 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:43.558959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:43.559250 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 20:18:43.559419 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 20:18:43.559699 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":94,"threads_started":1}
I20260620 20:18:43.561923 11505 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33291
I20260620 20:18:43.562116 11505 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:43.562505 11505 heartbeater.cc:511] Master 127.2.235.254:33291 requested a full tablet report, sending...
I20260620 20:18:43.563381 11179 ts_manager.cc:194] Registered new tserver with Master: ed7c8fba7f4245c6aeb0ee6c2a6db5cd (127.2.235.194:44799)
I20260620 20:18:43.563874 11179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:44291
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:18:43.579360 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33291
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:43.676960 11510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:43.677162 11510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:43.677201 11510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:43.680961 11510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:43.681007 11510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:43.681083 11510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:43.685868 11510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:43.686697 11510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:43.687546 11510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:43.695657 11516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.695662 11515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.695847 11510 server_base.cc:1034] running on GCE node
W20260620 20:18:43.695932 11518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.696300 11510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:43.696875 11510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:43.698041 11510 hybrid_clock.cc:648] HybridClock initialized: now 1781986723698014 us; error 51 us; skew 500 ppm
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:43.701007 11510 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:18:43.701721 11510 webserver.cc:460] Webserver started at http://127.2.235.195:40959/ using document root <none> and password file <none>
I20260620 20:18:43.702198 11510 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:43.702278 11510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:43.702457 11510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:43.704772 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "c7a832f987704984b48343de7b63b016"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "0e517511ecd49e121f131bf1382440a9"
server_key_iv: "bce3839da153df904f2639832242c53d"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.705524 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "c7a832f987704984b48343de7b63b016"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "0e517511ecd49e121f131bf1382440a9"
server_key_iv: "bce3839da153df904f2639832242c53d"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.710146 11510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:18:43.713352 11525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.714795 11510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:18:43.714905 11510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "c7a832f987704984b48343de7b63b016"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "0e517511ecd49e121f131bf1382440a9"
server_key_iv: "bce3839da153df904f2639832242c53d"
server_key_version: "encryptionkey@0"
I20260620 20:18:43.714993 11510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:43.750752 11510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:43.753324 11510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:43.753495 11510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:43.754024 11510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:43.754725 11510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:43.754765 11510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.754823 11510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:43.754863 11510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:43.762784 11510 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:46627
I20260620 20:18:43.762818 11638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:46627 every 8 connection(s)
I20260620 20:18:43.763521 11510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:43.764328 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11510
I20260620 20:18:43.764444 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:18:43.764731 2991 external_mini_cluster.cc:1428] Setting key 247b5f3bc6feb438353931db120e6a83
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:18:43.775467 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.765129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:46305 (local address 127.2.235.254:33291)
0620 20:18:43.765300 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:43.765305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:43.766130 (+ 825us) server_negotiation.cc:407] Connection header received
0620 20:18:43.767432 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:43.767436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:43.767488 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:43.767562 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:43.768596 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.769147 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.770159 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.770379 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.770490 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.772473 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:43.772476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:43.772479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:43.772505 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:43.773689 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.774132 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.774137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.774138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.774213 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:43.774518 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:43.774521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:43.774523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:43.774684 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:43.774788 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:43.775206 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 20:18:43.775311 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:18:43.776099 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:43.765507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:46305)
0620 20:18:43.766011 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:43.766062 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:43.767241 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:43.767719 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:43.767726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:43.768033 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:43.768426 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:43.768437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.769277 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.769280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:43.770043 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:43.770051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:43.770520 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:43.770524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:43.770672 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:43.771080 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:43.771093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:43.772337 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:43.773796 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:43.773800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:43.773812 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:43.774036 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:43.774297 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:43.774300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:43.774302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:43.774423 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:43.774787 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:43.774793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:43.775002 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 20:18:43.775173 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 20:18:43.775438 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":142,"threads_started":1}
I20260620 20:18:43.777107 11639 heartbeater.cc:347] Connected to a master server at 127.2.235.254:33291
I20260620 20:18:43.777364 11639 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:43.777871 11639 heartbeater.cc:511] Master 127.2.235.254:33291 requested a full tablet report, sending...
I20260620 20:18:43.778743 11179 ts_manager.cc:194] Registered new tserver with Master: c7a832f987704984b48343de7b63b016 (127.2.235.195:46627)
I20260620 20:18:43.779242 11179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:46305
I20260620 20:18:43.789963 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab.
Entry for principal oryx/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab.
WARNING: no policy specified for oryx/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab.
Entry for principal oryx/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab.
WARNING: no policy specified for oryx/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab.
Entry for principal oryx/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab.
I20260620 20:18:43.851310 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11241
I20260620 20:18:43.856650 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11376
I20260620 20:18:43.861542 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11510
I20260620 20:18:43.865989 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11147
I20260620 20:18:43.871963 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33291
--webserver_interface=127.2.235.254
--webserver_port=33619
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:33291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:43.971527 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:43.971763 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:43.971841 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:43.975453 11648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:43.975522 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:43.975558 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:43.975607 11648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:43.975643 11648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:43.980341 11648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:33291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=33619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:43.981144 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:43.981971 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:43.989228 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:43.989344 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.989464 11648 server_base.cc:1034] running on GCE node
W20260620 20:18:43.989636 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:43.990082 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:43.990921 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:43.992072 11648 hybrid_clock.cc:648] HybridClock initialized: now 1781986723992058 us; error 24 us; skew 500 ppm
Jun 20 20:18:43 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986723, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:43.994436 11648 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:18:43.995126 11648 webserver.cc:460] Webserver started at http://127.2.235.254:33619/ using document root <none> and password file <none>
I20260620 20:18:43.995568 11648 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:43.995649 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:43.998852 11648 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 20:18:44.000560 11663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.001597 11648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:44.001729 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "8b3f328d5dfa4b6287e17c6be261f466"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "92b4ecfb36f75dcb62e217e46977e15e"
server_key_iv: "fe3c97f953eda036f252c3dd5a783a6d"
server_key_version: "encryptionkey@0"
I20260620 20:18:44.002157 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:44.021147 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:44.023789 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:44.023904 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:44.029719 11648 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:33291
I20260620 20:18:44.029772 11715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:33291 every 8 connection(s)
I20260620 20:18:44.030633 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:44.034292 11716 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:18:44.035637 11716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Bootstrap starting.
I20260620 20:18:44.036741 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11648
I20260620 20:18:44.037225 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:32973
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:33291
--webserver_port=39533
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:44.050966 11716 log.cc:824] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:44.056365 11716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:18:44.056715 11716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Bootstrap complete.
I20260620 20:18:44.060401 11716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:44.060736 11716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3f328d5dfa4b6287e17c6be261f466, State: Initialized, Role: FOLLOWER
I20260620 20:18:44.061182 11716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:44.061316 11716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:44.061393 11716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:44.061512 11716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:18:44.063102 11716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:44.063386 11716 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b3f328d5dfa4b6287e17c6be261f466; no voters:
I20260620 20:18:44.063658 11716 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:18:44.063724 11720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:18:44.063899 11720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [term 2 LEADER]: Becoming Leader. State: Replica: 8b3f328d5dfa4b6287e17c6be261f466, State: Running, Role: LEADER
I20260620 20:18:44.064146 11720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } }
I20260620 20:18:44.064697 11716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:44.064941 11722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3f328d5dfa4b6287e17c6be261f466. Latest consensus state: current_term: 2 leader_uuid: "8b3f328d5dfa4b6287e17c6be261f466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } } }
I20260620 20:18:44.065241 11722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:44.065141 11721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8b3f328d5dfa4b6287e17c6be261f466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3f328d5dfa4b6287e17c6be261f466" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33291 } } }
I20260620 20:18:44.065623 11721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466 [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:44.066404 11726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:44.069159 11726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:44.069768 11726 catalog_manager.cc:1259] Loaded cluster ID: 395f9baad97d443b8d8638ac94f3fdb2
I20260620 20:18:44.069826 11726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:44.072027 11726 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:44.072652 11726 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8b3f328d5dfa4b6287e17c6be261f466: Loaded TSK: 0
I20260620 20:18:44.073745 11726 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:18:44.136726 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:44.136963 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:44.137032 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:44.140672 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:44.140748 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:44.140827 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:44.145171 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.193.keytab
--principal=oryx/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:32973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=39533
--tserver_master_addrs=127.2.235.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:44.145937 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:44.146759 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:44.155586 11742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:44.155757 11718 server_base.cc:1034] running on GCE node
W20260620 20:18:44.155591 11743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:44.155622 11745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:44.156322 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:44.156941 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:44.158113 11718 hybrid_clock.cc:648] HybridClock initialized: now 1781986724158081 us; error 51 us; skew 500 ppm
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986724, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:44.160347 11718 init.cc:377] Logged in from keytab as oryx/127.2.235.193@KRBTEST.COM (short username oryx)
I20260620 20:18:44.161043 11718 webserver.cc:460] Webserver started at http://127.2.235.193:39533/ using document root <none> and password file <none>
I20260620 20:18:44.161525 11718 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:44.161586 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:44.164992 11718 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:44.167244 11752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.168373 11718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:18:44.168486 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "3762c7a4139b49f288097241c875e4a0"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "950f648e1426b1496332e0de309fefac"
server_key_iv: "c2460410344c0357686106d249d4330b"
server_key_version: "encryptionkey@0"
I20260620 20:18:44.168943 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:44.182845 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:44.185700 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:44.185847 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:44.186349 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:44.187132 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:44.187186 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.187222 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:44.187275 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.194955 11718 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:32973
I20260620 20:18:44.194978 11865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:32973 every 8 connection(s)
I20260620 20:18:44.195744 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:44.202876 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11718
I20260620 20:18:44.203289 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:44799
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:33291
--webserver_port=33075
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:18:44.207531 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.197537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:33921 (local address 127.2.235.254:33291)
0620 20:18:44.197948 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.197963 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.198493 (+ 530us) server_negotiation.cc:407] Connection header received
0620 20:18:44.199365 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.199374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.199607 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.199832 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.200689 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.201695 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.202851 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.203318 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.203508 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.206881 (+ 3373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:33921: BlockingRecv error: recv got EOF from 127.2.235.193:33921 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":113,"threads_started":1}
I20260620 20:18:44.207648 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.197721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:33921)
0620 20:18:44.198352 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.198396 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.199070 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.199906 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.199910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.200179 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.200559 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.200570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.201813 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.201816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.202737 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.202750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.203563 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.203568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.203729 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.204227 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.204240 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.206720 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":123,"threads_started":1}
W20260620 20:18:44.208069 11866 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:33291 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.235.254:33291: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.212793 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.208451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:34671)
0620 20:18:44.208525 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.208538 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.208618 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.208831 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.208833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.208964 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.209033 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.209037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.209688 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.209693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.210143 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.210150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.210551 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.210554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.210635 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.211039 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.211049 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.212677 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 20:18:44.212874 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.208544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:34671 (local address 127.2.235.254:33291)
0620 20:18:44.208662 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.208665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.208677 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:44.208705 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.208708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.208741 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.208820 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.209126 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.209598 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.210244 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.210453 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.210549 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.212775 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:34671: BlockingRecv error: recv got EOF from 127.2.235.193:34671 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.217464 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.213184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:56167)
0620 20:18:44.213287 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.213307 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.213382 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.213655 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.213657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.213777 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.213835 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.213839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.214440 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.214444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.214896 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.214903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.215276 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.215279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.215348 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.215737 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.215746 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.217341 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 20:18:44.217607 11866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:18:44.217547 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.213432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:56167 (local address 127.2.235.254:33291)
0620 20:18:44.213511 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.213514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.213522 (+ 8us) server_negotiation.cc:407] Connection header received
0620 20:18:44.213544 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.213545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.213572 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.213642 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.213929 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.214347 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.215001 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.215182 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.215275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.217448 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:56167: BlockingRecv error: recv got EOF from 127.2.235.193:56167 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 20:18:44.301946 11870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:44.302137 11870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:44.302201 11870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:44.305683 11870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:44.305727 11870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:44.305806 11870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:18:44.310640 11870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.194.keytab
--principal=oryx/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:44799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=33075
--tserver_master_addrs=127.2.235.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:44.311417 11870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:44.312179 11870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:44.320279 11875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:44.320271 11878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:44.320271 11876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:44.320585 11870 server_base.cc:1034] running on GCE node
I20260620 20:18:44.321043 11870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:44.321666 11870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:44.322842 11870 hybrid_clock.cc:648] HybridClock initialized: now 1781986724322811 us; error 46 us; skew 500 ppm
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986724, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:44.325133 11870 init.cc:377] Logged in from keytab as oryx/127.2.235.194@KRBTEST.COM (short username oryx)
I20260620 20:18:44.325896 11870 webserver.cc:460] Webserver started at http://127.2.235.194:33075/ using document root <none> and password file <none>
I20260620 20:18:44.326376 11870 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:44.326481 11870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:44.329824 11870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:44.331701 11885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.332727 11870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:44.332849 11870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "ed7c8fba7f4245c6aeb0ee6c2a6db5cd"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "71effa6b85fc1c54468b11817b81ea11"
server_key_iv: "9b21dea509829a875e0fb84800f54b41"
server_key_version: "encryptionkey@0"
I20260620 20:18:44.333365 11870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:44.352332 11870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:44.354728 11870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:44.354889 11870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:44.355351 11870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:44.356021 11870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:44.356091 11870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.356168 11870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:44.356215 11870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.364073 11870 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:44799
I20260620 20:18:44.364092 11998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:44799 every 8 connection(s)
I20260620 20:18:44.365000 11870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:18:44.369453 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 11870
I20260620 20:18:44.369972 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:46627
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:33291
--webserver_port=40959
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:18:44.375986 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.366755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:38591 (local address 127.2.235.254:33291)
0620 20:18:44.366904 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.366908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.367838 (+ 930us) server_negotiation.cc:407] Connection header received
0620 20:18:44.368539 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.368541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.368586 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.368650 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.369911 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.370509 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.371758 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.372097 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.372196 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.375877 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:38591: BlockingRecv error: recv got EOF from 127.2.235.194:38591 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:18:44.376515 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.367096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:38591)
0620 20:18:44.367712 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.367746 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.368387 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.368810 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.368815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.369113 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.369747 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.369766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.370626 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.370631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.371648 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.371656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.372245 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.372251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.372495 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.373003 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.373027 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.375766 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":101,"threads_started":1}
W20260620 20:18:44.377122 11999 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:33291 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.235.254:33291: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.381747 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.377565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:48549)
0620 20:18:44.377685 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.377699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.377791 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.377998 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.378001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.378129 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.378197 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.378201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.378798 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.378804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.379259 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.379266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.379671 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.379674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.379761 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.380165 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.380175 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.381632 (+ 1457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:18:44.381844 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.377690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:48549 (local address 127.2.235.254:33291)
0620 20:18:44.377826 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.377831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.377844 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:44.377883 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.377884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.377914 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.377987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.378303 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.378703 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.379362 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.379568 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.379661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.381738 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:48549: BlockingRecv error: recv got EOF from 127.2.235.194:48549 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.386766 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.382338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:45751)
0620 20:18:44.382556 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.382568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.382667 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.382902 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.382905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.383050 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.383118 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.383122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.383800 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.383803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.384242 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.384248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.384623 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.384626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.384693 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.385089 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.385100 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.386657 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161}
W20260620 20:18:44.386930 11999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:18:44.386961 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.382450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:45751 (local address 127.2.235.254:33291)
0620 20:18:44.382538 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.382542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.382634 (+ 92us) server_negotiation.cc:407] Connection header received
0620 20:18:44.382767 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.382770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.382810 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.382889 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.383230 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.383657 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.384344 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.384528 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.384624 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.386760 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:45751: BlockingRecv error: recv got EOF from 127.2.235.194:45751 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 20:18:44.469197 12002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:44.469455 12002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:44.469491 12002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:44.473215 12002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:44.473271 12002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:44.473354 12002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:18:44.478189 12002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/krb5kdc/oryx_127.2.235.195.keytab
--principal=oryx/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:46627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=40959
--tserver_master_addrs=127.2.235.254:33291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:44.478982 12002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:44.479796 12002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:44.488268 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:44.488298 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:44.488284 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:44.488324 12002 server_base.cc:1034] running on GCE node
I20260620 20:18:44.488844 12002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:44.489400 12002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:44.490571 12002 hybrid_clock.cc:648] HybridClock initialized: now 1781986724490551 us; error 31 us; skew 500 ppm
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986724, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:44.493093 12002 init.cc:377] Logged in from keytab as oryx/127.2.235.195@KRBTEST.COM (short username oryx)
I20260620 20:18:44.493929 12002 webserver.cc:460] Webserver started at http://127.2.235.195:40959/ using document root <none> and password file <none>
I20260620 20:18:44.494390 12002 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:44.494459 12002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:44.497851 12002 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:44.499529 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.500510 12002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:18:44.500622 12002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "c7a832f987704984b48343de7b63b016"
format_stamp: "Formatted at 2026-06-20 20:18:43 on dist-test-slave-8z53"
server_key: "0e517511ecd49e121f131bf1382440a9"
server_key_iv: "bce3839da153df904f2639832242c53d"
server_key_version: "encryptionkey@0"
I20260620 20:18:44.501046 12002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:44.516727 12002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:44.519695 12002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:44.519882 12002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:44.520429 12002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:44.521139 12002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:44.521209 12002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.521296 12002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:44.521340 12002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:44.529793 12002 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:46627
I20260620 20:18:44.529858 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:46627 every 8 connection(s)
I20260620 20:18:44.530788 12002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:18:44.535692 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12002
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:18:44.540948 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.532372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:44199 (local address 127.2.235.254:33291)
0620 20:18:44.532500 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.532504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.533469 (+ 965us) server_negotiation.cc:407] Connection header received
0620 20:18:44.534170 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.534172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.534217 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.534287 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.535156 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.535805 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.537292 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.537584 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.537704 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.540840 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:44199: BlockingRecv error: recv got EOF from 127.2.235.195:44199 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:18:44.541507 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.532716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:44199)
0620 20:18:44.533345 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.533382 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.534009 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.534416 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.534422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.534686 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.535042 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.535053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.535920 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.535922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.537132 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.537149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.537746 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.537750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.537913 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.538387 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.538400 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.540741 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":112,"threads_started":1}
W20260620 20:18:44.541910 12131 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:33291 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.235.254:33291: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.546298 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.536679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.2.235.254:33291)
0620 20:18:44.536943 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.536955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.536966 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:44.536999 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.537001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.537036 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.537138 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.537953 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.538575 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.539795 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.540077 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.540178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.541805 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:44.541812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:44.541826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:44.541854 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:44.543877 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:44.544262 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:44.544268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:44.544273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:44.544430 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:44.544982 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:44.544987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:44.544988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:44.545455 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:44.545664 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:44.545892 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:18:44.546100 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.547132 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.542291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:47777)
0620 20:18:44.542391 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.542405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.542477 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.542789 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.542793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.542948 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.543076 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.543087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.543702 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.543708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.544416 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.544426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.544775 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.544777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.544838 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.545159 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.545173 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.547055 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 20:18:44.547307 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.542336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:47777 (local address 127.2.235.254:33291)
0620 20:18:44.542434 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.542439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.542453 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:44.542561 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.542565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.542605 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.542678 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.543190 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.543597 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.544519 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.544690 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.544753 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.547204 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:47777: BlockingRecv error: recv got EOF from 127.2.235.195:47777 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:44 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:44.552210 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:44.547666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:43617)
0620 20:18:44.547761 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:44.547775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:44.547868 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:44.548103 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:44.548106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:44.548252 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:44.548339 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.548347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.549122 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.549125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.549648 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:44.549656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.550073 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.550075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:44.550127 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.550522 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:44.550534 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:44.552084 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 20:18:44.552284 11868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:44.547743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:43617 (local address 127.2.235.254:33291)
0620 20:18:44.547884 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:44.547890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:44.547908 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:18:44.547955 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:44.547958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:44.547996 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:44.548078 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:44.548439 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.549005 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.549782 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:44.549981 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:44.550059 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:44.552183 (+ 2124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:43617: BlockingRecv error: recv got EOF from 127.2.235.195:43617 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:18:44.552385 12131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 20:18:45 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:45 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:45.226703 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:45.218106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:35915)
0620 20:18:45.218380 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:45.218393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:45.218470 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:45.218728 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:45.218730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:45.218883 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:45.218988 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:45.218993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.219721 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.219723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:45.220275 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:45.220281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.220682 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.220684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:45.220750 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:45.224650 (+ 3900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:45.224692 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:45.226578 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
W20260620 20:18:45.226841 12137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:45.218301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:35915 (local address 127.2.235.254:33291)
0620 20:18:45.218527 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:45.218530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:45.218541 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:45.218572 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:45.218574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:45.218610 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:45.218682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:45.219081 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.219615 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.220363 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.220569 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.220648 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:45.226716 (+ 6068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:35915: BlockingRecv error: recv got EOF from 127.2.235.193:35915 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
Jun 20 20:18:45 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:45 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:45.392344 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:45.387446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:37687)
0620 20:18:45.387759 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:45.387772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:45.387890 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:45.388135 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:45.388138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:45.388316 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:45.388397 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:45.388402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.389079 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.389084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:45.389678 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:45.389686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.390104 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.390107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:45.390187 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:45.390656 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:45.390666 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:45.392204 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
W20260620 20:18:45.392402 12137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:45.387558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:37687 (local address 127.2.235.254:33291)
0620 20:18:45.387700 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:45.387705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:45.387849 (+ 144us) server_negotiation.cc:407] Connection header received
0620 20:18:45.387983 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:45.387986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:45.388036 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:45.388114 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:45.388501 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.388982 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.389789 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.389995 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.390092 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:45.392304 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:37687: BlockingRecv error: recv got EOF from 127.2.235.194:37687 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 20:18:45 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:45 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:45.558020 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:45.552925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:58387)
0620 20:18:45.553167 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:45.553178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:45.553306 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:45.553554 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:45.553557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:45.553739 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:45.553833 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:45.553838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.554500 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.554505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:45.554996 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:45.555002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.555413 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.555416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:45.555502 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:45.555997 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:45.556007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:45.557922 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260620 20:18:45.558184 12137 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:45.552964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:58387 (local address 127.2.235.254:33291)
0620 20:18:45.553116 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:45.553120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:45.553260 (+ 140us) server_negotiation.cc:407] Connection header received
0620 20:18:45.553396 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:45.553398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:45.553444 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:45.553537 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:45.553936 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.554406 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.555101 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:45.555316 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:45.555412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:45.558034 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:58387: BlockingRecv error: recv got EOF from 127.2.235.195:58387 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 20:18:46 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:46 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:46.232426 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:46.227424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:41303)
0620 20:18:46.227721 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:46.227738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:46.227844 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:46.228065 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:46.228066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:46.228220 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:46.228309 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:46.228314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.229035 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.229038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:46.229610 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:46.229619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.230013 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.230015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:46.230071 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:46.230600 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:46.230613 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:46.232318 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
W20260620 20:18:46.232535 12141 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:46.227483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:41303 (local address 127.2.235.254:33291)
0620 20:18:46.227732 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:46.227736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:46.227803 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:18:46.227950 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:46.227951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:46.227992 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:46.228053 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:46.228397 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.228890 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.229726 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.229922 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.229992 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:46.232425 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:41303: BlockingRecv error: recv got EOF from 127.2.235.193:41303 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
Jun 20 20:18:46 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:46 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:46.398443 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:46.393032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:51831)
0620 20:18:46.393327 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:46.393339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:46.393437 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:46.393681 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:46.393684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:46.393858 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:46.393941 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:46.393946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.394630 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.394634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:46.395143 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:46.395150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.395677 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.395681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:46.395769 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:46.396315 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:46.396333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:46.398271 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
W20260620 20:18:46.398473 12141 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:46.393135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:51831 (local address 127.2.235.254:33291)
0620 20:18:46.393298 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:46.393302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:46.393404 (+ 102us) server_negotiation.cc:407] Connection header received
0620 20:18:46.393510 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:46.393513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:46.393562 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:46.393642 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:46.394047 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.394524 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.395261 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.395547 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.395655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:46.398384 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:51831: BlockingRecv error: recv got EOF from 127.2.235.194:51831 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:18:46 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:46 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:46.563208 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:46.558673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:33257)
0620 20:18:46.558910 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:46.558923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:46.558999 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:46.559220 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:46.559222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:46.559380 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:46.559471 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:46.559476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.560130 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.560132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:46.560628 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:46.560633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.560962 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.560964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:46.561016 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:46.561503 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:46.561514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:46.563114 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
W20260620 20:18:46.563337 12141 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:46.558827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:33257 (local address 127.2.235.254:33291)
0620 20:18:46.558950 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:46.558954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:46.558966 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:46.559076 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:46.559079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:46.559121 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:46.559189 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:46.559573 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.560050 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.560707 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:46.560883 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:46.560943 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:46.563228 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:33257: BlockingRecv error: recv got EOF from 127.2.235.195:33257 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 20:18:47 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:47 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:47.238181 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:47.233198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:42795)
0620 20:18:47.233466 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:47.233479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:47.233558 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:47.233774 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:47.233775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:47.233924 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:47.234010 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:47.234015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.234723 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.234726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:47.235293 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:47.235300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.235699 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.235701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:47.235781 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:47.236221 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:47.236231 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:47.238069 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
W20260620 20:18:47.238329 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:47.233241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:42795 (local address 127.2.235.254:33291)
0620 20:18:47.233521 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:47.233524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:47.233535 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:47.233639 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:47.233641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:47.233679 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:47.233744 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:47.234089 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.234630 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.235384 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.235614 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.235678 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:47.238182 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:42795: BlockingRecv error: recv got EOF from 127.2.235.193:42795 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
Jun 20 20:18:47 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:47 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:47.403669 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:47.399050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:47675)
0620 20:18:47.399300 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:47.399311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:47.399389 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:47.399570 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:47.399573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:47.399719 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:47.399803 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:47.399807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.400408 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.400410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:47.400875 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:47.400881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.401216 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.401218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:47.401318 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:47.401788 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:47.401799 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:47.403534 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":108,"threads_started":1}
W20260620 20:18:47.403793 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:47.399099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:47675 (local address 127.2.235.254:33291)
0620 20:18:47.399225 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:47.399229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:47.399373 (+ 144us) server_negotiation.cc:407] Connection header received
0620 20:18:47.399445 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:47.399447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:47.399485 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:47.399540 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:47.399884 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.400334 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.400958 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.401137 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.401201 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:47.403650 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:47675: BlockingRecv error: recv got EOF from 127.2.235.194:47675 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 20:18:47 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:47 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:47.569095 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:47.563825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:56337)
0620 20:18:47.564066 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:47.564079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:47.564162 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:47.564387 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:47.564389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:47.564574 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:47.564665 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:47.564671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.565445 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.565448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:47.566114 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:47.566123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.566733 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.566736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:47.566850 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:47.567341 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:47.567352 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:47.568985 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
W20260620 20:18:47.569232 12145 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:47.563975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:56337 (local address 127.2.235.254:33291)
0620 20:18:47.564098 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:47.564103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:47.564119 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:47.564242 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:47.564245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:47.564288 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:47.564359 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:47.564756 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.565328 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.566281 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:47.566587 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:47.566743 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:47.569118 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:56337: BlockingRecv error: recv got EOF from 127.2.235.195:56337 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 20:18:48 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:48 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:48.243960 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:48.238846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:36959)
0620 20:18:48.239085 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:48.239097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:48.239177 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:48.239384 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:48.239386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:48.239534 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:48.239626 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:48.239631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.240350 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.240353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:48.240880 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:48.240888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.241357 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.241360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:48.241438 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:48.241922 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:48.241933 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:48.243857 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
W20260620 20:18:48.244074 12149 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:48.238892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:36959 (local address 127.2.235.254:33291)
0620 20:18:48.239136 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:48.239139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:48.239150 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:48.239253 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:48.239255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:48.239291 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:48.239355 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:48.239715 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.240256 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.241007 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.241237 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.241341 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:48.243966 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:36959: BlockingRecv error: recv got EOF from 127.2.235.193:36959 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
Jun 20 20:18:48 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:48 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:48.409746 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:48.404439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:60617)
0620 20:18:48.404682 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:48.404697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:48.404786 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:48.405000 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:48.405003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:48.405176 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:48.405336 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:48.405343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.406145 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.406151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:48.406741 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:48.406750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.407138 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.407141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:48.407203 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:48.407670 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:48.407681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:48.409585 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260620 20:18:48.409781 12149 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:48.404470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:60617 (local address 127.2.235.254:33291)
0620 20:18:48.404617 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:48.404622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:48.404759 (+ 137us) server_negotiation.cc:407] Connection header received
0620 20:18:48.404842 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:48.404844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:48.404882 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:48.404941 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:48.405457 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.406017 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.406837 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.407043 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.407129 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:48.409691 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:60617: BlockingRecv error: recv got EOF from 127.2.235.194:60617 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 20:18:48 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:48 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:48.575007 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:48.569883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:43783)
0620 20:18:48.570153 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:48.570166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:48.570255 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:48.570463 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:48.570465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:48.570643 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:48.570766 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:48.570777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.571581 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.571583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:48.572208 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:48.572217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.572650 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.572653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:48.572715 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:48.573325 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:48.573335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:48.574904 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
W20260620 20:18:48.575194 12149 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:48.569916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:43783 (local address 127.2.235.254:33291)
0620 20:18:48.570039 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:48.570042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:48.570212 (+ 170us) server_negotiation.cc:407] Connection header received
0620 20:18:48.570331 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:48.570334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:48.570376 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:48.570432 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:48.570884 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.571494 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.572310 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:48.572561 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:48.572643 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:48.575054 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:43783: BlockingRecv error: recv got EOF from 127.2.235.195:43783 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 20:18:49 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:49 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:49.249791 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:49.244681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:46399)
0620 20:18:49.244963 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:49.244976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:49.245052 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:49.245300 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:49.245303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:49.245476 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:49.245579 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:49.245584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.246279 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.246282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:49.246815 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:49.246823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.247307 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.247310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:49.247372 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:49.247847 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:49.247858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:49.249686 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
W20260620 20:18:49.249943 12153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:49.244854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:46399 (local address 127.2.235.254:33291)
0620 20:18:49.245041 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:49.245044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:49.245055 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:18:49.245117 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:49.245119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:49.245155 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:49.245223 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:49.245701 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.246184 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.246960 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.247199 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.247292 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:49.249812 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:46399: BlockingRecv error: recv got EOF from 127.2.235.193:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
Jun 20 20:18:49 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:49 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:49.416276 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:49.410422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:51793)
0620 20:18:49.410804 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:49.410819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:49.410928 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:49.411206 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:49.411209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:49.411395 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:49.411479 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:49.411484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.412221 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.412226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:49.412790 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:49.412798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.413249 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.413253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:49.413361 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:49.414104 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:49.414127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:49.416100 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":87,"threads_started":1}
W20260620 20:18:49.416364 12153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:49.410597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:51793 (local address 127.2.235.254:33291)
0620 20:18:49.410740 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:49.410745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:49.410917 (+ 172us) server_negotiation.cc:407] Connection header received
0620 20:18:49.411044 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:49.411046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:49.411094 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:49.411179 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:49.411590 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.412115 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.412908 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.413136 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.413250 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:49.416235 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:51793: BlockingRecv error: recv got EOF from 127.2.235.194:51793 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 20:18:49 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:49 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:49.581004 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:49.575682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:34809)
0620 20:18:49.575988 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:49.576000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:49.576085 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:49.576303 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:49.576305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:49.576480 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:49.576568 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:49.576573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.577343 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.577347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:49.577925 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:49.577933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.578392 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.578394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:49.578449 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:49.578954 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:49.578965 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:49.580898 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
W20260620 20:18:49.581144 12153 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:49.575786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:34809 (local address 127.2.235.254:33291)
0620 20:18:49.575921 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:49.575925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:49.576064 (+ 139us) server_negotiation.cc:407] Connection header received
0620 20:18:49.576167 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:49.576169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:49.576213 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:49.576270 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:49.576689 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.577218 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.578024 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:49.578281 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:49.578375 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:49.581020 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:34809: BlockingRecv error: recv got EOF from 127.2.235.195:34809 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 20:18:50 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:50 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:50.260340 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:50.250484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:56953)
0620 20:18:50.250748 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:50.250761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:50.250859 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:50.251135 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:50.251137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:50.251322 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:50.251414 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:50.251419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.252173 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.252177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:50.252731 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:50.252739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.253132 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.253134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:50.253191 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:50.258544 (+ 5353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:50.258556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:50.260233 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
W20260620 20:18:50.260516 12157 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:50.250568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:56953 (local address 127.2.235.254:33291)
0620 20:18:50.250809 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:50.250813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:50.250825 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:50.250954 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:50.250956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:50.250998 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:50.251068 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:50.251519 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.252068 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.252827 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.253040 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.253114 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:50.260364 (+ 7250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:56953: BlockingRecv error: recv got EOF from 127.2.235.193:56953 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
Jun 20 20:18:50 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:50 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:50.421664 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:50.416933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:56849)
0620 20:18:50.417194 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:50.417207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:50.417322 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:50.417538 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:50.417541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:50.417728 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:50.417820 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:50.417825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.418544 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.418546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:50.419067 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:50.419074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.419452 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.419454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:50.419505 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:50.419950 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:50.419960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:50.421542 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
W20260620 20:18:50.421790 12157 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:50.417028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:56849 (local address 127.2.235.254:33291)
0620 20:18:50.417190 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:50.417193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:50.417263 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:18:50.417401 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:50.417404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:50.417443 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:50.417500 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:50.417917 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.418453 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.419181 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.419371 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.419438 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:50.421683 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:56849: BlockingRecv error: recv got EOF from 127.2.235.194:56849 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 20:18:50 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:50 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:50.586862 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:50.582005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:53221)
0620 20:18:50.582232 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:50.582245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:50.582353 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:50.582613 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:50.582615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:50.582774 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:50.582861 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:50.582867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.583648 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.583650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:50.584186 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:50.584194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.584558 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.584561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:50.584628 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:50.585130 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:50.585141 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:50.586762 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
W20260620 20:18:50.587109 12157 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:50.582034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:53221 (local address 127.2.235.254:33291)
0620 20:18:50.582154 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:50.582159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:50.582321 (+ 162us) server_negotiation.cc:407] Connection header received
0620 20:18:50.582442 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:50.582444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:50.582489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:50.582550 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:50.582975 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.583540 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.584280 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:50.584474 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:50.584544 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:50.586876 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:53221: BlockingRecv error: recv got EOF from 127.2.235.195:53221 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 20:18:51 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:51 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:51.265970 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:51.261015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:50415)
0620 20:18:51.261369 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:51.261384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:51.261475 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:51.261718 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:51.261720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:51.261890 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:51.261993 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:51.261998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.262657 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.262660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:51.263161 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:51.263168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.263532 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.263534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:51.263606 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.264067 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:51.264077 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:51.265870 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":81,"threads_started":1}
W20260620 20:18:51.266144 12160 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:51.261071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50415 (local address 127.2.235.254:33291)
0620 20:18:51.261337 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:51.261340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:51.261447 (+ 107us) server_negotiation.cc:407] Connection header received
0620 20:18:51.261562 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:51.261564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:51.261609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:51.261681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:51.262079 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.262563 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.263255 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.263446 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.263510 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.265972 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:50415: BlockingRecv error: recv got EOF from 127.2.235.193:50415 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
Jun 20 20:18:51 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:51 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:51.427007 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:51.422354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:59217)
0620 20:18:51.422633 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:51.422644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:51.422730 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:51.422925 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:51.422927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:51.423077 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:51.423169 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:51.423174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.423780 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.423782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:51.424419 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:51.424428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.424869 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.424874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:51.424954 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.425394 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:51.425403 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:51.426907 (+ 1504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
W20260620 20:18:51.427187 12160 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:51.422407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:59217 (local address 127.2.235.254:33291)
0620 20:18:51.422544 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:51.422548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:51.422709 (+ 161us) server_negotiation.cc:407] Connection header received
0620 20:18:51.422801 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:51.422802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:51.422841 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:51.422897 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:51.423250 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.423697 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.424519 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.424751 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.424865 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.427020 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:59217: BlockingRecv error: recv got EOF from 127.2.235.194:59217 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 20:18:51 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:51 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:51.592777 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:51.587544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:48241)
0620 20:18:51.587855 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:51.587869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:51.587962 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:51.588197 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:51.588199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:51.588369 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:51.588460 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:51.588465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.589196 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.589199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:51.589781 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:51.589789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.590168 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.590170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:51.590227 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.590778 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:51.590790 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:51.592677 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
W20260620 20:18:51.592936 12160 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:51.587600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:48241 (local address 127.2.235.254:33291)
0620 20:18:51.587741 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:51.587744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:51.587937 (+ 193us) server_negotiation.cc:407] Connection header received
0620 20:18:51.588040 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:51.588043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:51.588094 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:51.588162 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:51.588554 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.589107 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.589869 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:51.590086 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:51.590153 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:51.592796 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:48241: BlockingRecv error: recv got EOF from 127.2.235.195:48241 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:18:52 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:52 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:52.272330 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:52.266734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:40747)
0620 20:18:52.267023 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:52.267037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:52.267136 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:52.267382 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:52.267384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:52.267548 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:52.267643 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:52.267648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.268670 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.268674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:52.269588 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:52.269598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.270005 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.270007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:52.270062 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:52.270551 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:52.270561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:52.272228 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
W20260620 20:18:52.272511 12165 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:52.266866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:40747 (local address 127.2.235.254:33291)
0620 20:18:52.267112 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:52.267115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:52.267129 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:52.267225 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:52.267227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:52.267271 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:52.267351 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:52.267756 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.268558 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.269694 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.269914 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.269992 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:52.272374 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:40747: BlockingRecv error: recv got EOF from 127.2.235.193:40747 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
Jun 20 20:18:52 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:52 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:52.432940 12166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:52.427723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:37315)
0620 20:18:52.428010 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:52.428023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:52.428122 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:52.428377 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:52.428380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:52.428608 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:52.428699 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:52.428704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.429605 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.429607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:52.430126 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:52.430133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.430503 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.430506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:52.430574 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:52.431008 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:52.431017 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:52.432772 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
W20260620 20:18:52.432998 12165 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:52.427771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:37315 (local address 127.2.235.254:33291)
0620 20:18:52.427919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:52.427924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:52.428097 (+ 173us) server_negotiation.cc:407] Connection header received
0620 20:18:52.428200 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:52.428202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:52.428248 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:52.428311 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:52.428793 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.429499 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.430217 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.430424 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.430488 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:52.432891 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:37315: BlockingRecv error: recv got EOF from 127.2.235.194:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 20:18:52 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:52 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:52.598742 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:52.593490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:53751)
0620 20:18:52.593817 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:52.593830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:52.593928 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:52.594203 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:52.594206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:52.594401 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:52.594509 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:52.594515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.595214 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.595218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:52.595718 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:52.595726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.596144 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.596147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:52.596232 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:52.596779 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:52.596790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:52.598636 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
W20260620 20:18:52.598989 12165 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:52.593605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:53751 (local address 127.2.235.254:33291)
0620 20:18:52.593752 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:52.593756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:52.593885 (+ 129us) server_negotiation.cc:407] Connection header received
0620 20:18:52.594034 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:52.594037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:52.594093 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:52.594185 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:52.594625 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.595104 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.595832 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:52.596037 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:52.596142 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:52.598858 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:53751: BlockingRecv error: recv got EOF from 127.2.235.195:53751 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 20:18:53 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:53 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:53.278488 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:53.273100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:35265)
0620 20:18:53.273417 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:53.273430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:53.273512 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:53.273768 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:53.273770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:53.273941 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:53.274035 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.274040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.274805 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.274807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.275357 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.275364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.275823 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.275826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.275891 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.276402 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:53.276413 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:53.278373 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
W20260620 20:18:53.278782 12169 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:53.273153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:35265 (local address 127.2.235.254:33291)
0620 20:18:53.273455 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:53.273458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:53.273474 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:53.273603 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:53.273606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:53.273653 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:53.273737 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:53.274157 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.274708 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.275483 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.275718 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.275805 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.278523 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:35265: BlockingRecv error: recv got EOF from 127.2.235.193:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":117,"threads_started":1}
Jun 20 20:18:53 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:53 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:53.439504 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:53.433722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:38001)
0620 20:18:53.433981 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:53.433995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:53.434084 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:53.434358 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:53.434360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:53.434528 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:53.434647 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.434653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.435419 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.435422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.436089 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.436098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.436633 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.436636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.436698 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.437194 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:53.437208 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:53.439373 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
W20260620 20:18:53.439674 12169 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:53.433789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:38001 (local address 127.2.235.254:33291)
0620 20:18:53.433962 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:53.433966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:53.434066 (+ 100us) server_negotiation.cc:407] Connection header received
0620 20:18:53.434171 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:53.434173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:53.434219 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:53.434293 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:53.434770 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.435309 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.436206 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.436505 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.436615 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.439514 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:38001: BlockingRecv error: recv got EOF from 127.2.235.194:38001 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 20:18:53 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:53 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:53.604174 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:53.599414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:43803)
0620 20:18:53.599637 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:53.599652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:53.599743 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:53.600032 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:53.600034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:53.600202 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:53.600288 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.600293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.600937 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.600939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.601546 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:53.601554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.601927 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.601929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:53.601983 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.602403 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:53.602413 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:53.604063 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
W20260620 20:18:53.604331 12169 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:53.599455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:43803 (local address 127.2.235.254:33291)
0620 20:18:53.599575 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:53.599584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:53.599715 (+ 131us) server_negotiation.cc:407] Connection header received
0620 20:18:53.599876 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:53.599879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:53.599927 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:53.600006 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:53.600387 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.600856 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.601640 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:53.601831 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:53.601909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:53.604172 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:43803: BlockingRecv error: recv got EOF from 127.2.235.195:43803 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 20:18:54 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:54 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:54.284488 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.279339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:38983)
0620 20:18:54.279614 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:54.279630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:54.279735 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:54.279980 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:54.279982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:54.280175 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:54.280260 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.280265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.280947 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.280959 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.281477 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.281484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.281936 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.281939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.282018 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.282564 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:54.282584 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:54.284318 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
W20260620 20:18:54.284548 12173 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:54.279390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:38983 (local address 127.2.235.254:33291)
0620 20:18:54.279614 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:54.279619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:54.279689 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:18:54.279822 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:54.279825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:54.279875 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:54.279961 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:54.280367 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.280854 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.281618 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.281836 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.281935 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.284429 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:38983: BlockingRecv error: recv got EOF from 127.2.235.193:38983 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
Jun 20 20:18:54 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:54 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:54.445323 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.440160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:52013)
0620 20:18:54.440485 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:54.440505 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:54.440649 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:54.440904 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:54.440907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:54.441122 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:54.441209 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.441213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.441941 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.441944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.442470 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.442476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.442908 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.442911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.442992 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.443428 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:54.443438 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:54.445164 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":85,"threads_started":1}
W20260620 20:18:54.445456 12173 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:54.440289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:52013 (local address 127.2.235.254:33291)
0620 20:18:54.440454 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:54.440458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:54.440590 (+ 132us) server_negotiation.cc:407] Connection header received
0620 20:18:54.440745 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:54.440749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:54.440796 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:54.440887 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:54.441351 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.441836 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.442593 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.442809 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.442907 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.445325 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:52013: BlockingRecv error: recv got EOF from 127.2.235.194:52013 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 20:18:54 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:54 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:54.610507 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:54.605017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:50951)
0620 20:18:54.605273 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:54.605287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:54.605373 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:54.605637 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:54.605640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:54.605826 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:54.605990 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.606001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.606874 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.606878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.607445 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:54.607453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.607914 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.607918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:54.608003 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.608531 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:54.608551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:54.610344 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
W20260620 20:18:54.610545 12173 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:54.605173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:50951 (local address 127.2.235.254:33291)
0620 20:18:54.605331 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:54.605335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:54.605348 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:54.605483 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:54.605488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:54.605535 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:54.605612 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:54.606125 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.606775 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.607562 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:54.607805 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:54.607911 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:54.610446 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:50951: BlockingRecv error: recv got EOF from 127.2.235.195:50951 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:18:55 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:55 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:55.294230 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:55.285216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:51341)
0620 20:18:55.285506 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:55.285519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:55.285612 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:55.285846 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:55.285848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:55.286007 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:55.286098 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:55.286103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.286824 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.286826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:55.287369 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:55.287376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.287820 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.287823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:55.287891 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.292430 (+ 4539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:55.292440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:55.294110 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
W20260620 20:18:55.294363 12177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:55.285278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:51341 (local address 127.2.235.254:33291)
0620 20:18:55.285484 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:55.285487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:55.285588 (+ 101us) server_negotiation.cc:407] Connection header received
0620 20:18:55.285685 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:55.285687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:55.285729 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:55.285812 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:55.286217 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.286724 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.287490 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.287717 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.287802 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.294246 (+ 6444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:51341: BlockingRecv error: recv got EOF from 127.2.235.193:51341 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
Jun 20 20:18:55 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:55 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:55.451331 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:55.445986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:39607)
0620 20:18:55.446297 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:55.446315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:55.446442 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:55.446727 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:55.446730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:55.446919 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:55.447004 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:55.447009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.447732 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.447739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:55.448307 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:55.448314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.448774 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.448778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:55.448856 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.449397 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:55.449408 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:55.451221 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
W20260620 20:18:55.451452 12177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:55.446087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:39607 (local address 127.2.235.254:33291)
0620 20:18:55.446235 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:55.446257 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:55.446385 (+ 128us) server_negotiation.cc:407] Connection header received
0620 20:18:55.446564 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:55.446566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:55.446610 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:55.446695 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:55.447113 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.447629 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.448425 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.448669 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.448771 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.451342 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:39607: BlockingRecv error: recv got EOF from 127.2.235.194:39607 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:18:55 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:55 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:55.615820 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:55.611223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:50757)
0620 20:18:55.611454 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:55.611466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:55.611559 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:55.611759 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:55.611760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:55.611911 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:55.612001 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:55.612005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.612650 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.612652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:55.613141 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:55.613147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.613538 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.613540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:55.613604 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.614061 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:55.614071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:55.615691 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
W20260620 20:18:55.615937 12177 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:55.611259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:50757 (local address 127.2.235.254:33291)
0620 20:18:55.611386 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:55.611391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:55.611534 (+ 143us) server_negotiation.cc:407] Connection header received
0620 20:18:55.611633 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:55.611634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:55.611677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:55.611734 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:55.612085 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.612571 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.613222 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:55.613453 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:55.613521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:55.615827 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:50757: BlockingRecv error: recv got EOF from 127.2.235.195:50757 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 20:18:56 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:56 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:56.299628 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:56.294970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:46523)
0620 20:18:56.295205 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:56.295216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:56.295300 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:56.295510 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:56.295512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:56.295672 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:56.295760 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.295764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.296407 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.296409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.296904 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.296910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.297316 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.297319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.297378 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.297874 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:56.297885 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:56.299524 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
W20260620 20:18:56.299790 12181 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:56.295006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:46523 (local address 127.2.235.254:33291)
0620 20:18:56.295226 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:56.295230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:56.295260 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:18:56.295372 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:56.295374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:56.295411 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:56.295477 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:56.295844 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.296326 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.296989 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.297185 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.297301 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.299635 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:46523: BlockingRecv error: recv got EOF from 127.2.235.193:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
Jun 20 20:18:56 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:56 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:56.456655 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:56.452008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:45683)
0620 20:18:56.452257 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:56.452269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:56.452352 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:56.452580 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:56.452582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:56.452740 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:56.452831 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.452836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.453557 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.453559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.454051 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.454058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.454402 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.454404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.454454 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.454921 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:56.454931 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:56.456545 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260620 20:18:56.456775 12181 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:56.452122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:45683 (local address 127.2.235.254:33291)
0620 20:18:56.452250 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:56.452257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:56.452330 (+ 73us) server_negotiation.cc:407] Connection header received
0620 20:18:56.452411 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:56.452413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:56.452451 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:56.452511 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:56.452921 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.453440 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.454139 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.454322 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.454387 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.456666 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:45683: BlockingRecv error: recv got EOF from 127.2.235.194:45683 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 20:18:56 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:56 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:56.622104 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:56.616550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:46047)
0620 20:18:56.616872 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:56.616894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:56.617034 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:56.617392 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:56.617397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:56.617645 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:56.617770 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.617777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.618786 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.618789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.619346 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:56.619354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.619783 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.619785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:56.619839 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.620276 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:56.620286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:56.621986 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
W20260620 20:18:56.622236 12181 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:56.616646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:46047 (local address 127.2.235.254:33291)
0620 20:18:56.616832 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:56.616837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:56.616967 (+ 130us) server_negotiation.cc:407] Connection header received
0620 20:18:56.617131 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:56.617135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:56.617189 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:56.617305 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:56.617894 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.618670 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.619446 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:56.619677 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:56.619771 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:56.622115 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:46047: BlockingRecv error: recv got EOF from 127.2.235.195:46047 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 20:18:57 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:57 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:57.310097 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:57.300362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:42209)
0620 20:18:57.300728 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:57.300743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:57.300849 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:57.301076 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:57.301078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:57.301321 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:57.301418 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.301425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.302177 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.302180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.302736 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.302746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.303350 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.303353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.303419 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.304069 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:57.304087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:57.309850 (+ 5763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
W20260620 20:18:57.310176 12185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:57.300373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:42209 (local address 127.2.235.254:33291)
0620 20:18:57.300694 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:57.300697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:57.300798 (+ 101us) server_negotiation.cc:407] Connection header received
0620 20:18:57.300916 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:57.300918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:57.300957 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:57.301040 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:57.301527 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.302066 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.302859 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.303205 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.303319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.310026 (+ 6707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:42209: BlockingRecv error: recv got EOF from 127.2.235.193:42209 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":127,"threads_started":1}
Jun 20 20:18:57 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:57 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:57.463620 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:57.457438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:59253)
0620 20:18:57.457805 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:57.457822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:57.457950 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:57.458274 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:57.458277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:57.458511 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:57.458616 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.458622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.459766 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.459768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.460272 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.460280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.460916 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.460918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.460978 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.461598 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:57.461609 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:57.463497 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":100,"threads_started":1}
W20260620 20:18:57.463842 12185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:57.457581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:59253 (local address 127.2.235.254:33291)
0620 20:18:57.457741 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:57.457746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:57.457913 (+ 167us) server_negotiation.cc:407] Connection header received
0620 20:18:57.458068 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:57.458073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:57.458141 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:57.458224 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:57.458739 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.459630 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.460373 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.460804 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.460918 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.463691 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:59253: BlockingRecv error: recv got EOF from 127.2.235.194:59253 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:18:57 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:57 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:57.627928 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:57.622747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:52643)
0620 20:18:57.623012 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:57.623028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:57.623127 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:57.623346 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:57.623348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:57.623514 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:57.623624 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.623629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.624319 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.624321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.624868 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:57.624875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.625299 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.625303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:57.625376 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.625841 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:57.625853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:57.627816 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
W20260620 20:18:57.628057 12185 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:57.622855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:52643 (local address 127.2.235.254:33291)
0620 20:18:57.622998 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:57.623002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:57.623100 (+ 98us) server_negotiation.cc:407] Connection header received
0620 20:18:57.623207 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:57.623209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:57.623252 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:57.623311 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:57.623717 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.624228 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.624961 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:57.625161 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:57.625232 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:57.627935 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:52643: BlockingRecv error: recv got EOF from 127.2.235.195:52643 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:18:58 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:58 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:58.317829 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.312804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:51009)
0620 20:18:58.313084 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.313096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.313176 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.313424 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.313426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.313595 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:58.313694 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.313699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.314363 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.314365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.314869 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.314875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.315231 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.315233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.315286 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.315807 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:58.315826 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:58.317619 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
W20260620 20:18:58.317883 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:58.312855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:51009 (local address 127.2.235.254:33291)
0620 20:18:58.313158 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.313161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.313173 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:58.313277 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.313279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.313321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.313389 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.313779 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.314277 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.314952 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.315148 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.315213 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.317745 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:51009: BlockingRecv error: recv got EOF from 127.2.235.193:51009 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":55,"threads_started":1}
Jun 20 20:18:58 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:58 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:58.469980 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.464330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:33211)
0620 20:18:58.464578 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.464592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.464689 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.464932 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.464934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.465155 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:58.465292 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.465300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.466056 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.466059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.466662 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.466670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.467065 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.467067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.467146 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.467805 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:58.467832 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:58.469815 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":83,"threads_started":1}
W20260620 20:18:58.470026 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:58.464462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:33211 (local address 127.2.235.254:33291)
0620 20:18:58.464590 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.464599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.464658 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:18:58.464761 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.464763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.464802 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.464860 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.465427 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.465962 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.466756 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.466935 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.467003 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.469915 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:33211: BlockingRecv error: recv got EOF from 127.2.235.194:33211 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 20:18:58 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:58 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.195@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:58.633185 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:58.628600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.195:46553)
0620 20:18:58.628873 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:58.628887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:58.629005 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:58.629214 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:58.629216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:58.629423 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:58.629531 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.629537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.630180 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.630181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.630673 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:58.630679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.631012 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.631014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:58.631066 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.631521 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:58.631536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:58.633086 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":112,"threads_started":1}
W20260620 20:18:58.633320 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:58.628667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:46553 (local address 127.2.235.254:33291)
0620 20:18:58.628813 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:58.628818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:58.628979 (+ 161us) server_negotiation.cc:407] Connection header received
0620 20:18:58.629079 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:58.629081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:58.629124 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:58.629181 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:58.629623 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.630102 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.630755 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:58.630936 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:58.630999 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:58.633198 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.195:46553: BlockingRecv error: recv got EOF from 127.2.235.195:46553 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 20:18:59 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:59 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.193@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:59.323603 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.318623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.193:57857)
0620 20:18:59.318870 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:59.318883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:59.318978 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:59.319187 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:59.319189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:59.319340 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:59.319421 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:59.319426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.320097 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.320099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:59.320632 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:59.320638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.321033 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.321035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:59.321089 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.321580 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:59.321595 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:59.323422 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
W20260620 20:18:59.323678 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:59.318728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:57857 (local address 127.2.235.254:33291)
0620 20:18:59.318970 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:59.318973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:59.318993 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:18:59.319056 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:59.319057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:59.319092 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:59.319155 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:59.319512 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.320013 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.320724 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.320946 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.321020 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.323558 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.193:57857: BlockingRecv error: recv got EOF from 127.2.235.193:57857 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
Jun 20 20:18:59 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:18:59 dist-test-slave-8z53 krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.235.194@KRBTEST.COM for oryx/127.2.235.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:18:59.475916 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.470721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:33291 (local address 127.2.235.194:59943)
0620 20:18:59.471104 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:59.471125 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:59.471244 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:59.471486 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:59.471488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:59.471695 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:59.471780 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:59.471785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.472573 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.472578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:59.473152 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:59.473160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.473624 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.473627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:59.473707 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.474145 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:59.474155 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:59.475775 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.235.254:33291: Server oryx/127.2.235.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":75,"threads_started":1}
W20260620 20:18:59.475970 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:59.470813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:59943 (local address 127.2.235.254:33291)
0620 20:18:59.471082 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:59.471087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:59.471192 (+ 105us) server_negotiation.cc:407] Connection header received
0620 20:18:59.471338 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:59.471343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:59.471390 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:59.471468 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:59.471910 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.472468 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.473290 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.473515 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.473614 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.475878 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.235.194:59943: BlockingRecv error: recv got EOF from 127.2.235.194:59943 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 20:18:59.545713 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11718
I20260620 20:18:59.551112 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11870
I20260620 20:18:59.555842 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12002
I20260620 20:18:59.560426 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 11648
2026-06-20T20:18:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17696 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T20:18:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:59Z Disabled control of system clock
I20260620 20:18:59.588033 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:39959
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:18:59.688217 12198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:59.688422 12198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:59.688453 12198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:59.692095 12198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:59.692135 12198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:59.692148 12198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:59.692162 12198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:59.692191 12198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:59.696205 12198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:39959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:59.697032 12198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:59.698295 12198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:59.705864 12203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:59.705876 12204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:59.706058 12198 server_base.cc:1034] running on GCE node
W20260620 20:18:59.706192 12206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:59.706710 12198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:59.707624 12198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:59.708779 12198 hybrid_clock.cc:648] HybridClock initialized: now 1781986739708758 us; error 30 us; skew 500 ppm
I20260620 20:18:59.710429 12198 webserver.cc:460] Webserver started at http://127.2.235.254:46775/ using document root <none> and password file <none>
I20260620 20:18:59.710948 12198 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:59.711004 12198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:59.711227 12198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:59.712951 12198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "0b467eaface5403abb1c834646c691ef"
format_stamp: "Formatted at 2026-06-20 20:18:59 on dist-test-slave-8z53"
server_key: "5b0f0556d4a0e8a3c0df271d0e677e7c"
server_key_iv: "2fc9c7d4392bd6da25ec5a88be428359"
server_key_version: "encryptionkey@0"
I20260620 20:18:59.713486 12198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "0b467eaface5403abb1c834646c691ef"
format_stamp: "Formatted at 2026-06-20 20:18:59 on dist-test-slave-8z53"
server_key: "5b0f0556d4a0e8a3c0df271d0e677e7c"
server_key_iv: "2fc9c7d4392bd6da25ec5a88be428359"
server_key_version: "encryptionkey@0"
I20260620 20:18:59.716609 12198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:18:59.718827 12212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:59.719887 12198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:59.720016 12198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "0b467eaface5403abb1c834646c691ef"
format_stamp: "Formatted at 2026-06-20 20:18:59 on dist-test-slave-8z53"
server_key: "5b0f0556d4a0e8a3c0df271d0e677e7c"
server_key_iv: "2fc9c7d4392bd6da25ec5a88be428359"
server_key_version: "encryptionkey@0"
I20260620 20:18:59.720117 12198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:59.739301 12198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:59.739890 12198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:59.740069 12198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:59.746439 12198 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:39959
I20260620 20:18:59.746441 12264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:39959 every 8 connection(s)
I20260620 20:18:59.747249 12198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:59.749810 12265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:59.752588 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12198
I20260620 20:18:59.752703 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:18:59.752983 2991 external_mini_cluster.cc:1428] Setting key 71252f7cfe8ac289eaf50d37244d5456
I20260620 20:18:59.754907 12265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef: Bootstrap starting.
I20260620 20:18:59.757339 12265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:59.758132 12265 log.cc:824] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:59.759999 12265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef: No bootstrap required, opened a new log
I20260620 20:18:59.762629 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.754273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.2.235.254:39959)
0620 20:18:59.754823 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:59.754841 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:59.754876 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:18:59.755638 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:59.755658 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:59.756085 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:59.756420 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:59.756815 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.758260 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.759265 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.759889 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.760027 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.760806 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:59.760812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:59.760825 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:59.761266 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:59.761336 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:59.761362 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:59.761453 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:59.761591 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 20:18:59.761831 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
I20260620 20:18:59.764160 12265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b467eaface5403abb1c834646c691ef" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39959 } }
I20260620 20:18:59.764333 12265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:18:59.764387 12265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b467eaface5403abb1c834646c691ef, State: Initialized, Role: FOLLOWER
I20260620 20:18:59.764777 12265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b467eaface5403abb1c834646c691ef" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39959 } }
I20260620 20:18:59.764883 12265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:18:59.764927 12265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:18:59.764987 12265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:59.765708 12265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b467eaface5403abb1c834646c691ef" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39959 } }
I20260620 20:18:59.765950 12265 leader_election.cc:302] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b467eaface5403abb1c834646c691ef; no voters:
I20260620 20:18:59.766153 12265 leader_election.cc:288] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:18:59.766249 12270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:59.766465 12270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [term 1 LEADER]: Becoming Leader. State: Replica: 0b467eaface5403abb1c834646c691ef, State: Running, Role: LEADER
I20260620 20:18:59.766793 12270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b467eaface5403abb1c834646c691ef" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39959 } }
I20260620 20:18:59.767197 12265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:59.768267 12271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b467eaface5403abb1c834646c691ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b467eaface5403abb1c834646c691ef" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39959 } } }
I20260620 20:18:59.768405 12271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:59.768555 12272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b467eaface5403abb1c834646c691ef. Latest consensus state: current_term: 1 leader_uuid: "0b467eaface5403abb1c834646c691ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b467eaface5403abb1c834646c691ef" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 39959 } } }
I20260620 20:18:59.768723 12272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:59.769553 12281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:59.771888 12281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:18:59.776314 12281 catalog_manager.cc:1347] Generated new cluster ID: b066b9b7b1aa4fc29900f8d2d984a1ee
I20260620 20:18:59.776376 12281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:59.788244 12281 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:59.789002 12281 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:59.793087 12281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b467eaface5403abb1c834646c691ef: Generated new TSK 0
I20260620 20:18:59.793635 12281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:18:59.803570 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39959
--builtin_ntp_servers=127.2.235.212:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 20:18:59.902493 12289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:59.902681 12289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:59.902712 12289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:59.906275 12289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:59.906316 12289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:59.906360 12289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:18:59.910494 12289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:59.911245 12289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:59.912328 12289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:59.920816 12295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:59.920840 12297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:59.920908 12289 server_base.cc:1034] running on GCE node
W20260620 20:18:59.920858 12294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:59.921419 12289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:59.921941 12289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:59.923115 12289 hybrid_clock.cc:648] HybridClock initialized: now 1781986739923102 us; error 22 us; skew 500 ppm
I20260620 20:18:59.925158 12289 webserver.cc:460] Webserver started at http://127.2.235.193:46843/ using document root <none> and password file <none>
I20260620 20:18:59.925758 12289 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:59.925815 12289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:59.926035 12289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:59.927763 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "5357a869d39d448494c322f60f40e888"
format_stamp: "Formatted at 2026-06-20 20:18:59 on dist-test-slave-8z53"
server_key: "8624a09a70edb3f3c3c92d4e336020d0"
server_key_iv: "39b2ed0de8836e6aa077cc1539eabf2a"
server_key_version: "encryptionkey@0"
I20260620 20:18:59.928355 12289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "5357a869d39d448494c322f60f40e888"
format_stamp: "Formatted at 2026-06-20 20:18:59 on dist-test-slave-8z53"
server_key: "8624a09a70edb3f3c3c92d4e336020d0"
server_key_iv: "39b2ed0de8836e6aa077cc1539eabf2a"
server_key_version: "encryptionkey@0"
I20260620 20:18:59.931811 12289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:18:59.934204 12303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:59.935472 12289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:18:59.935640 12289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "5357a869d39d448494c322f60f40e888"
format_stamp: "Formatted at 2026-06-20 20:18:59 on dist-test-slave-8z53"
server_key: "8624a09a70edb3f3c3c92d4e336020d0"
server_key_iv: "39b2ed0de8836e6aa077cc1539eabf2a"
server_key_version: "encryptionkey@0"
I20260620 20:18:59.935751 12289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:59.955480 12289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:59.956092 12289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:59.956280 12289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:59.956800 12289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:18:59.957670 12289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:18:59.957741 12289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:59.957803 12289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:18:59.957858 12289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:59.965807 12289 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:39483
I20260620 20:18:59.965842 12416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:39483 every 8 connection(s)
I20260620 20:18:59.966557 12289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:18:59.968577 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12289
I20260620 20:18:59.968699 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:18:59.969044 2991 external_mini_cluster.cc:1428] Setting key ac0e8ab05ac799d9e9e30764194a0afa
I20260620 20:18:59.970920 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39959
--builtin_ntp_servers=127.2.235.212:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:18:59.975725 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.968214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:58627 (local address 127.2.235.254:39959)
0620 20:18:59.968372 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:59.968377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:59.969619 (+ 1242us) server_negotiation.cc:407] Connection header received
0620 20:18:59.971006 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:59.971009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:59.971060 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:59.971130 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:59.971902 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.972422 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.973838 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.974051 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.974130 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.975299 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:59.975302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:59.975304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:18:59.975314 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:59.975331 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:18:59.975338 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:59.975399 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:59.975580 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 20:18:59.975627 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:18:59.976817 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:59.968737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:39959 (local address 127.2.235.193:58627)
0620 20:18:59.969469 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:59.969538 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:59.970822 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:59.971279 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:59.971285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:59.971357 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:59.971784 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:59.971794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.972543 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.972546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:59.973710 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:59.973722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:59.974191 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:59.974197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:59.974404 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:59.974989 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:18:59.974992 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:59.975041 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:18:59.975115 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:59.975425 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:59.975435 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:59.975442 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:18:59.975576 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:18:59.975854 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":135,"threads_started":1}
I20260620 20:18:59.977943 12417 heartbeater.cc:347] Connected to a master server at 127.2.235.254:39959
I20260620 20:18:59.978171 12417 heartbeater.cc:464] Registering TS with master...
I20260620 20:18:59.978621 12417 heartbeater.cc:511] Master 127.2.235.254:39959 requested a full tablet report, sending...
I20260620 20:18:59.980048 12229 ts_manager.cc:194] Registered new tserver with Master: 5357a869d39d448494c322f60f40e888 (127.2.235.193:39483)
I20260620 20:18:59.981138 12229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.193:58627
W20260620 20:19:00.070436 12420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:00.070650 12420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:00.070709 12420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:00.074366 12420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:00.074437 12420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:00.074522 12420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:19:00.078675 12420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:00.079507 12420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:00.080564 12420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:00.088583 12425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.088589 12426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.088624 12420 server_base.cc:1034] running on GCE node
W20260620 20:19:00.088589 12428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.089224 12420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:00.089782 12420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:00.090936 12420 hybrid_clock.cc:648] HybridClock initialized: now 1781986740090914 us; error 28 us; skew 500 ppm
I20260620 20:19:00.092713 12420 webserver.cc:460] Webserver started at http://127.2.235.194:40475/ using document root <none> and password file <none>
I20260620 20:19:00.093199 12420 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:00.093327 12420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:00.093559 12420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:00.095356 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "3566972b74ce4c8190760f7713421ba7"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "8e7d87b37a9a5929f683abef55e075b8"
server_key_iv: "4a7b458776383fd65c25602c78c270d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.095907 12420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "3566972b74ce4c8190760f7713421ba7"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "8e7d87b37a9a5929f683abef55e075b8"
server_key_iv: "4a7b458776383fd65c25602c78c270d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.099416 12420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:19:00.101686 12434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.102689 12420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:19:00.102808 12420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "3566972b74ce4c8190760f7713421ba7"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "8e7d87b37a9a5929f683abef55e075b8"
server_key_iv: "4a7b458776383fd65c25602c78c270d1"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.102905 12420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:00.124241 12420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:00.124773 12420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:00.124976 12420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:00.125550 12420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:00.126336 12420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:00.126396 12420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.126449 12420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:00.126483 12420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.134016 12420 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:35983
I20260620 20:19:00.134037 12547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:35983 every 8 connection(s)
I20260620 20:19:00.134730 12420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:00.136457 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12420
I20260620 20:19:00.136606 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:19:00.136922 2991 external_mini_cluster.cc:1428] Setting key a457ad9950b07303dca981c57fca5f92
I20260620 20:19:00.138769 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39959
--builtin_ntp_servers=127.2.235.212:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:19:00.143258 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.136155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:59769 (local address 127.2.235.254:39959)
0620 20:19:00.136301 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:00.136305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:00.137452 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 20:19:00.138877 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:00.138880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:00.138935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:00.139036 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:00.139915 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.140505 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.141607 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.141820 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.141932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:00.142842 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:00.142846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:00.142848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:19:00.142858 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:00.142877 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:19:00.142887 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:00.142954 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:00.143091 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:19:00.143140 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:19:00.143867 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.136608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:39959 (local address 127.2.235.194:59769)
0620 20:19:00.137298 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:00.137345 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:00.138687 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:00.139196 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:00.139202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:00.139346 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:00.139786 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:00.139797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.140610 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.140613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:00.141493 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:00.141508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.141962 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.141965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:00.142097 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:00.142637 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:19:00.142639 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:00.142669 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:19:00.142717 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:00.142965 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:00.142974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:00.142979 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:19:00.143085 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:19:00.143294 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260620 20:19:00.144788 12548 heartbeater.cc:347] Connected to a master server at 127.2.235.254:39959
I20260620 20:19:00.144992 12548 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:00.145537 12548 heartbeater.cc:511] Master 127.2.235.254:39959 requested a full tablet report, sending...
I20260620 20:19:00.146462 12229 ts_manager.cc:194] Registered new tserver with Master: 3566972b74ce4c8190760f7713421ba7 (127.2.235.194:35983)
I20260620 20:19:00.146925 12229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.194:59769
W20260620 20:19:00.237224 12551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:00.237493 12551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:00.237524 12551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:00.241484 12551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:00.241523 12551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:00.241586 12551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:19:00.245796 12551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:39959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:00.246544 12551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:00.247627 12551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:00.255770 12557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.255798 12556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.255779 12559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.255987 12551 server_base.cc:1034] running on GCE node
I20260620 20:19:00.256397 12551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:00.256910 12551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:00.258091 12551 hybrid_clock.cc:648] HybridClock initialized: now 1781986740258082 us; error 26 us; skew 500 ppm
I20260620 20:19:00.259764 12551 webserver.cc:460] Webserver started at http://127.2.235.195:41053/ using document root <none> and password file <none>
I20260620 20:19:00.260285 12551 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:00.260367 12551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:00.260565 12551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:00.262477 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "8ed0a2085b624b0590e9835d2feaeaf6"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "c98da8f9107730df10574922225573ac"
server_key_iv: "933be7e5d56e7d0d7e6446dbbb255b02"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.262996 12551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "8ed0a2085b624b0590e9835d2feaeaf6"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "c98da8f9107730df10574922225573ac"
server_key_iv: "933be7e5d56e7d0d7e6446dbbb255b02"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.266083 12551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:19:00.268355 12565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.269534 12551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:19:00.269639 12551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "8ed0a2085b624b0590e9835d2feaeaf6"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "c98da8f9107730df10574922225573ac"
server_key_iv: "933be7e5d56e7d0d7e6446dbbb255b02"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.269728 12551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:00.288540 12551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:00.289099 12551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:00.289316 12551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:00.289790 12551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:00.290541 12551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:00.290603 12551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.290663 12551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:00.290704 12551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.298801 12551 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:36755
I20260620 20:19:00.298823 12678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:36755 every 8 connection(s)
I20260620 20:19:00.299535 12551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:00.304150 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12551
I20260620 20:19:00.304245 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:19:00.304500 2991 external_mini_cluster.cc:1428] Setting key e3a782d33a5d1af53a7d6308087f5986
I20260620 20:19:00.311781 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.301010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:57455 (local address 127.2.235.254:39959)
0620 20:19:00.301204 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:00.301208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:00.302065 (+ 857us) server_negotiation.cc:407] Connection header received
0620 20:19:00.303235 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:00.303237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:00.303277 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:00.303338 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:00.304054 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.304736 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.305986 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.306304 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.306401 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:00.311335 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:00.311339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:00.311341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:19:00.311359 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:00.311376 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:19:00.311383 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:00.311453 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:00.311601 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 20:19:00.311656 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 20:19:00.312484 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.301429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:39959 (local address 127.2.235.195:57455)
0620 20:19:00.301962 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:00.302002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:00.303081 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:00.303463 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:00.303468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:00.303534 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:00.303932 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:00.303942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.304860 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.304866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:00.305879 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:00.305888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:00.306428 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:00.306431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:00.306586 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:00.311070 (+ 4484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:19:00.311071 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:00.311127 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:19:00.311196 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:00.311454 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:00.311463 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:00.311470 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:19:00.311587 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:19:00.311811 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260620 20:19:00.313480 12679 heartbeater.cc:347] Connected to a master server at 127.2.235.254:39959
I20260620 20:19:00.313657 12679 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:00.314047 12679 heartbeater.cc:511] Master 127.2.235.254:39959 requested a full tablet report, sending...
I20260620 20:19:00.314864 12229 ts_manager.cc:194] Registered new tserver with Master: 8ed0a2085b624b0590e9835d2feaeaf6 (127.2.235.195:36755)
I20260620 20:19:00.315312 12229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.195:57455
I20260620 20:19:00.317772 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:19:00.320292 12268 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:19:00.319512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.2.235.254:39959)
0620 20:19:00.319590 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:00.319593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:00.319794 (+ 201us) server_negotiation.cc:407] Connection header received
0620 20:19:00.319870 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:00.319872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:00.319904 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:00.319954 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:00.320155 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51262: BlockingRecv error: recv got EOF from 127.0.0.1:51262 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 20:19:00.320395 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12289
I20260620 20:19:00.326006 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12420
I20260620 20:19:00.331351 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12551
I20260620 20:19:00.335943 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12198
2026-06-20T20:19:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (774 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T20:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:00Z Disabled control of system clock
I20260620 20:19:00.363170 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:37899
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:19:00.458846 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:00.459067 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:00.459115 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:00.462435 12690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:00.462471 12690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:00.462484 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:00.462499 12690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:00.462512 12690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:00.466373 12690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:37899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:00.467175 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:00.468307 12690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:00.475611 12695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.475708 12696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.475749 12690 server_base.cc:1034] running on GCE node
W20260620 20:19:00.475792 12698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.476390 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:00.477447 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:00.478662 12690 hybrid_clock.cc:648] HybridClock initialized: now 1781986740478633 us; error 27 us; skew 500 ppm
I20260620 20:19:00.480352 12690 webserver.cc:460] Webserver started at http://127.2.235.254:33423/ using document root <none> and password file <none>
I20260620 20:19:00.480837 12690 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:00.480890 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:00.481106 12690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:00.482821 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "bd546307ea0b4072ba7d76bc858d2bcd"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "780c3b30fcc1c5270ad69f4b2fd2c1d4"
server_key_iv: "aade8243b6efdd737e15af726881903c"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.483340 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "bd546307ea0b4072ba7d76bc858d2bcd"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "780c3b30fcc1c5270ad69f4b2fd2c1d4"
server_key_iv: "aade8243b6efdd737e15af726881903c"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.486490 12690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:19:00.488677 12704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.489765 12690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:19:00.489903 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "bd546307ea0b4072ba7d76bc858d2bcd"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "780c3b30fcc1c5270ad69f4b2fd2c1d4"
server_key_iv: "aade8243b6efdd737e15af726881903c"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.490005 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:00.494031 12690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:00.494520 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:00.494675 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:00.500820 12690 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:37899
I20260620 20:19:00.500828 12756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:37899 every 8 connection(s)
I20260620 20:19:00.501631 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:00.504407 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:00.507197 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12690
I20260620 20:19:00.507316 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:19:00.507567 2991 external_mini_cluster.cc:1428] Setting key 5226111ad6ebef0d20fcb56105f8ebfe
I20260620 20:19:00.509613 12757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd: Bootstrap starting.
I20260620 20:19:00.511756 12757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:00.512393 12757 log.cc:824] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:00.512411 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.508830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.2.235.254:37899)
0620 20:19:00.509439 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:00.509453 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:00.509484 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:19:00.510092 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:00.510108 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:00.510393 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:00.510623 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:00.511355 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:00.511359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:00.511369 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:19:00.511379 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:00.511426 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:19:00.511440 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:00.511520 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:00.511637 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 20:19:00.511857 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
I20260620 20:19:00.514158 12757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd: No bootstrap required, opened a new log
I20260620 20:19:00.517181 12757 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37899 } }
I20260620 20:19:00.517413 12757 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:00.517457 12757 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd546307ea0b4072ba7d76bc858d2bcd, State: Initialized, Role: FOLLOWER
I20260620 20:19:00.517812 12757 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37899 } }
I20260620 20:19:00.517940 12757 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:00.517983 12757 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:00.518049 12757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:00.518755 12757 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37899 } }
I20260620 20:19:00.519014 12757 leader_election.cc:302] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd546307ea0b4072ba7d76bc858d2bcd; no voters:
I20260620 20:19:00.519241 12757 leader_election.cc:288] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:19:00.519341 12762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:00.519562 12762 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [term 1 LEADER]: Becoming Leader. State: Replica: bd546307ea0b4072ba7d76bc858d2bcd, State: Running, Role: LEADER
I20260620 20:19:00.520094 12762 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37899 } }
I20260620 20:19:00.520392 12757 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:00.521487 12763 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37899 } } }
I20260620 20:19:00.521607 12763 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:00.521873 12764 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd546307ea0b4072ba7d76bc858d2bcd. Latest consensus state: current_term: 1 leader_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd546307ea0b4072ba7d76bc858d2bcd" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37899 } } }
I20260620 20:19:00.521962 12764 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:00.522085 12771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:00.524443 12771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:00.529008 12771 catalog_manager.cc:1347] Generated new cluster ID: f5a85479d2864459a0cf87d73d5505ed
I20260620 20:19:00.529067 12771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:00.542358 12771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:00.542604 12771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:00.552898 12771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd546307ea0b4072ba7d76bc858d2bcd: Generated new TSK 0
I20260620 20:19:00.553519 12771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:19:00.560149 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37899
--builtin_ntp_servers=127.2.235.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:19:00.660648 12781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:00.660840 12781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:00.660872 12781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:00.664486 12781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:00.664534 12781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:00.664583 12781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:19:00.668759 12781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:00.669615 12781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:00.670687 12781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:00.679207 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.679208 12786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.679355 12781 server_base.cc:1034] running on GCE node
W20260620 20:19:00.679454 12787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.679823 12781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:00.680382 12781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:00.681533 12781 hybrid_clock.cc:648] HybridClock initialized: now 1781986740681517 us; error 40 us; skew 500 ppm
I20260620 20:19:00.683106 12781 webserver.cc:460] Webserver started at http://127.2.235.193:34131/ using document root <none> and password file <none>
I20260620 20:19:00.683589 12781 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:00.683643 12781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:00.683885 12781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:00.685748 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "1ccb9964fb8542eba6df16f468bf5397"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "f46b99bba7cef8cb6c6bc78ef405eae3"
server_key_iv: "1fdde4c80a254bbb28d64829a1328cc6"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.686261 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "1ccb9964fb8542eba6df16f468bf5397"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "f46b99bba7cef8cb6c6bc78ef405eae3"
server_key_iv: "1fdde4c80a254bbb28d64829a1328cc6"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.689445 12781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:19:00.691424 12796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.692356 12781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:19:00.692469 12781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "1ccb9964fb8542eba6df16f468bf5397"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "f46b99bba7cef8cb6c6bc78ef405eae3"
server_key_iv: "1fdde4c80a254bbb28d64829a1328cc6"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.692576 12781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:00.696381 12781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:00.696877 12781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:00.697034 12781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:00.697546 12781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:00.698333 12781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:00.698401 12781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.698462 12781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:00.698513 12781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.706307 12781 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:38199
I20260620 20:19:00.706349 12909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:38199 every 8 connection(s)
I20260620 20:19:00.707022 12781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:00.712034 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.708518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:34005 (local address 127.2.235.254:37899)
0620 20:19:00.708654 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:00.708659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:00.709671 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 20:19:00.710733 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:00.710735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:00.710768 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:00.710832 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:00.711704 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:00.711707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:00.711710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:19:00.711719 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:00.711738 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:19:00.711745 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:00.711810 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:00.711912 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 20:19:00.711944 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:19:00.712730 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.708905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37899 (local address 127.2.235.193:34005)
0620 20:19:00.709556 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:00.709590 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:00.710576 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:00.710964 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:00.710970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:00.711041 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:00.711477 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:19:00.711478 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:00.711510 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:19:00.711581 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:00.711811 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:00.711817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:00.711820 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:19:00.711928 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:19:00.712108 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
I20260620 20:19:00.713769 12910 heartbeater.cc:347] Connected to a master server at 127.2.235.254:37899
I20260620 20:19:00.713951 12910 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:00.714284 12910 heartbeater.cc:511] Master 127.2.235.254:37899 requested a full tablet report, sending...
I20260620 20:19:00.715639 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12781
I20260620 20:19:00.715659 12721 ts_manager.cc:194] Registered new tserver with Master: 1ccb9964fb8542eba6df16f468bf5397 (127.2.235.193:38199)
I20260620 20:19:00.715754 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:19:00.716037 2991 external_mini_cluster.cc:1428] Setting key de41b3918de4d2e14641eda4de2fc0c9
I20260620 20:19:00.717952 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37899
--builtin_ntp_servers=127.2.235.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:19:00.815232 12913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:00.815428 12913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:00.815462 12913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:00.819267 12913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:00.819308 12913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:00.819396 12913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:19:00.823926 12913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:00.824743 12913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:00.825986 12913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:00.834277 12918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.834331 12913 server_base.cc:1034] running on GCE node
W20260620 20:19:00.834317 12919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.834280 12921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.834890 12913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:00.835421 12913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:00.836611 12913 hybrid_clock.cc:648] HybridClock initialized: now 1781986740836599 us; error 23 us; skew 500 ppm
I20260620 20:19:00.838306 12913 webserver.cc:460] Webserver started at http://127.2.235.194:46245/ using document root <none> and password file <none>
I20260620 20:19:00.838825 12913 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:00.838912 12913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:00.839119 12913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:00.840852 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "948d21f5f0234a41ad2029daa505ff4a"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "ca92abccb82123e79a0d684ba7ced4df"
server_key_iv: "0110681d04a883b1ac1870250afa7377"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.841426 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "948d21f5f0234a41ad2029daa505ff4a"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "ca92abccb82123e79a0d684ba7ced4df"
server_key_iv: "0110681d04a883b1ac1870250afa7377"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.844597 12913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:19:00.846789 12927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.847875 12913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:19:00.848022 12913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "948d21f5f0234a41ad2029daa505ff4a"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "ca92abccb82123e79a0d684ba7ced4df"
server_key_iv: "0110681d04a883b1ac1870250afa7377"
server_key_version: "encryptionkey@0"
I20260620 20:19:00.848124 12913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:00.852103 12913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:00.852607 12913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:00.852757 12913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:00.853195 12913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:00.853981 12913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:00.854048 12913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.854108 12913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:00.854156 12913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:00.863188 12913 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:45899
I20260620 20:19:00.863222 13040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:45899 every 8 connection(s)
I20260620 20:19:00.863925 12913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:00.869179 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.865503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:41187 (local address 127.2.235.254:37899)
0620 20:19:00.865635 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:00.865640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:00.866613 (+ 973us) server_negotiation.cc:407] Connection header received
0620 20:19:00.867765 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:00.867767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:00.867807 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:00.867875 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:00.868816 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:00.868819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:00.868821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:19:00.868831 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:00.868851 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:19:00.868859 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:00.868925 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:00.869025 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 20:19:00.869059 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:19:00.869880 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:00.865881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37899 (local address 127.2.235.194:41187)
0620 20:19:00.866486 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:00.866521 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:00.867594 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:00.868010 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:00.868016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:00.868094 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:00.868578 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:19:00.868580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:00.868615 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:19:00.868682 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:00.868916 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:00.868919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:00.868923 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:19:00.869048 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 20:19:00.869231 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260620 20:19:00.870805 13041 heartbeater.cc:347] Connected to a master server at 127.2.235.254:37899
I20260620 20:19:00.870976 13041 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:00.871270 13041 heartbeater.cc:511] Master 127.2.235.254:37899 requested a full tablet report, sending...
I20260620 20:19:00.872161 12720 ts_manager.cc:194] Registered new tserver with Master: 948d21f5f0234a41ad2029daa505ff4a (127.2.235.194:45899)
I20260620 20:19:00.872962 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 12913
I20260620 20:19:00.873072 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:19:00.873407 2991 external_mini_cluster.cc:1428] Setting key e0b881e6920b09cdb02742618de4fef5
I20260620 20:19:00.875077 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37899
--builtin_ntp_servers=127.2.235.212:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:19:00.976126 13044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:00.976356 13044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:00.976387 13044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:00.980118 13044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:00.980157 13044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:00.980233 13044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:19:00.984302 13044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:00.985088 13044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:00.986207 13044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:00.994289 13049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.994318 13050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:00.994318 13052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:00.994592 13044 server_base.cc:1034] running on GCE node
I20260620 20:19:00.994963 13044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:00.995486 13044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:00.996651 13044 hybrid_clock.cc:648] HybridClock initialized: now 1781986740996632 us; error 33 us; skew 500 ppm
I20260620 20:19:00.998713 13044 webserver.cc:460] Webserver started at http://127.2.235.195:32931/ using document root <none> and password file <none>
I20260620 20:19:00.999178 13044 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:00.999230 13044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:00.999365 13044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:01.001106 13044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "9cc64c768cc8409a96a646566d251aee"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "b0c9f243e98bc128d33fcfb8492a1068"
server_key_iv: "ed5e116ae854bec98929fd4af5625aca"
server_key_version: "encryptionkey@0"
I20260620 20:19:01.001667 13044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "9cc64c768cc8409a96a646566d251aee"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "b0c9f243e98bc128d33fcfb8492a1068"
server_key_iv: "ed5e116ae854bec98929fd4af5625aca"
server_key_version: "encryptionkey@0"
I20260620 20:19:01.004732 13044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:19:01.006984 13058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:01.008039 13044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:19:01.008148 13044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "9cc64c768cc8409a96a646566d251aee"
format_stamp: "Formatted at 2026-06-20 20:19:00 on dist-test-slave-8z53"
server_key: "b0c9f243e98bc128d33fcfb8492a1068"
server_key_iv: "ed5e116ae854bec98929fd4af5625aca"
server_key_version: "encryptionkey@0"
I20260620 20:19:01.008244 13044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:01.012425 13044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:01.012903 13044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:01.013058 13044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:01.013581 13044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:01.014374 13044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:01.014436 13044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:01.014501 13044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:01.014549 13044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:01.023023 13044 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:41539
I20260620 20:19:01.023092 13171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:41539 every 8 connection(s)
I20260620 20:19:01.023841 13044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:01.029136 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:01.025360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:53305 (local address 127.2.235.254:37899)
0620 20:19:01.025481 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:01.025484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:01.026380 (+ 896us) server_negotiation.cc:407] Connection header received
0620 20:19:01.027529 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:01.027532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:01.027572 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:01.027663 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:01.028681 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:01.028685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:01.028688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:19:01.028700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:01.028722 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:19:01.028731 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:01.028830 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:01.028967 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:19:01.029010 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:19:01.029968 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 13044
I20260620 20:19:01.029973 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:01.025690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37899 (local address 127.2.235.195:53305)
0620 20:19:01.026260 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:01.026292 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:01.027344 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:01.027797 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:01.027803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:01.027875 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:01.028445 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:19:01.028446 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:01.028480 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:19:01.028541 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:01.028816 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:01.028824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:01.028829 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:19:01.029000 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 20:19:01.029234 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260620 20:19:01.030074 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:19:01.030373 2991 external_mini_cluster.cc:1428] Setting key 9ae3d869c3a1eb02f915e59263003a42
I20260620 20:19:01.031301 13172 heartbeater.cc:347] Connected to a master server at 127.2.235.254:37899
I20260620 20:19:01.031553 13172 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:01.031883 13172 heartbeater.cc:511] Master 127.2.235.254:37899 requested a full tablet report, sending...
I20260620 20:19:01.032774 12720 ts_manager.cc:194] Registered new tserver with Master: 9cc64c768cc8409a96a646566d251aee (127.2.235.195:41539)
I20260620 20:19:01.043833 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:19:01.046270 12760 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:19:01.045538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.2.235.254:37899)
0620 20:19:01.045649 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:01.045653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:01.045784 (+ 131us) server_negotiation.cc:407] Connection header received
0620 20:19:01.045873 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:01.045875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:01.045899 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:01.045955 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:01.046168 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50230: BlockingRecv error: recv got EOF from 127.0.0.1:50230 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:19:01.046671 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12781
I20260620 20:19:01.052608 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12913
I20260620 20:19:01.057706 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 13044
I20260620 20:19:01.062844 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 12690
2026-06-20T20:19:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (726 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:01 dist-test-slave-8z53 krb5kdc[13182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:01 dist-test-slave-8z53 krb5kdc[13182](info): set up 2 sockets
Jun 20 20:19:01 dist-test-slave-8z53 krb5kdc[13182](info): commencing operation
krb5kdc: starting...
W20260620 20:19:02.026095 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:19:02.131585 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41099
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:02.231470 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:02.231714 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:02.231782 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:02.235487 13195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:02.235563 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:02.235600 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:02.235656 13195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:02.235693 13195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:02.240494 13195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:02.241353 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:02.242209 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:02.249531 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.249675 13195 server_base.cc:1034] running on GCE node
W20260620 20:19:02.249626 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.249841 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.250365 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:02.251309 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:02.252496 13195 hybrid_clock.cc:648] HybridClock initialized: now 1781986742252483 us; error 33 us; skew 500 ppm
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:02.255033 13195 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:19:02.255805 13195 webserver.cc:460] Webserver started at http://127.2.235.254:39723/ using document root <none> and password file <none>
I20260620 20:19:02.256309 13195 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:02.256392 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:02.256599 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:02.258443 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "848a95925d61456f913811e527116d21"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "96be28fbb5b6e569064122b92e3e9466"
server_key_iv: "ce5e96cde2baa364aa4ac01146445f47"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.258971 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "848a95925d61456f913811e527116d21"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "96be28fbb5b6e569064122b92e3e9466"
server_key_iv: "ce5e96cde2baa364aa4ac01146445f47"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.262228 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:19:02.264344 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.265463 13195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:19:02.265583 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "848a95925d61456f913811e527116d21"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "96be28fbb5b6e569064122b92e3e9466"
server_key_iv: "ce5e96cde2baa364aa4ac01146445f47"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.265681 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:02.281309 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:02.283869 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:02.284057 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:02.290541 13195 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:41099
I20260620 20:19:02.290809 13262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:41099 every 8 connection(s)
I20260620 20:19:02.291401 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:02.293983 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:02.296558 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 13195
I20260620 20:19:02.296695 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:19:02.296983 2991 external_mini_cluster.cc:1428] Setting key bc9402d19f9ccf432c6b08930414be4c
I20260620 20:19:02.299602 13263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21: Bootstrap starting.
I20260620 20:19:02.302037 13263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:02.302889 13263 log.cc:824] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:02.304634 13263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21: No bootstrap required, opened a new log
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:02.308495 13263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a95925d61456f913811e527116d21" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41099 } }
I20260620 20:19:02.308753 13263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:02.308817 13263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848a95925d61456f913811e527116d21, State: Initialized, Role: FOLLOWER
I20260620 20:19:02.309321 13263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a95925d61456f913811e527116d21" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41099 } }
I20260620 20:19:02.309480 13263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:02.309558 13263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:02.309662 13263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:02.310590 13263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a95925d61456f913811e527116d21" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41099 } }
I20260620 20:19:02.310945 13263 leader_election.cc:302] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 848a95925d61456f913811e527116d21; no voters:
I20260620 20:19:02.311172 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.298271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.2.235.254:41099)
0620 20:19:02.298654 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.298669 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.298702 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:19:02.299303 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.299317 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.299607 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.299892 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.300673 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.301651 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.302794 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.303239 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.303343 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.305722 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.305728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.305742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.305768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.308151 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.308542 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.308547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.308552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.308677 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.308973 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.308976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.308977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.309476 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.309682 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.309948 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.310222 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":129,"threads_started":1}
I20260620 20:19:02.311244 13263 leader_election.cc:288] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:19:02.311369 13268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:02.311566 13268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [term 1 LEADER]: Becoming Leader. State: Replica: 848a95925d61456f913811e527116d21, State: Running, Role: LEADER
I20260620 20:19:02.311879 13268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a95925d61456f913811e527116d21" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41099 } }
I20260620 20:19:02.312341 13263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:02.312987 13268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 848a95925d61456f913811e527116d21. Latest consensus state: current_term: 1 leader_uuid: "848a95925d61456f913811e527116d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a95925d61456f913811e527116d21" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41099 } } }
I20260620 20:19:02.313129 13268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:02.313556 13276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:02.313556 13269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "848a95925d61456f913811e527116d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a95925d61456f913811e527116d21" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41099 } } }
I20260620 20:19:02.313766 13269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:02.316577 13276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:02.321619 13276 catalog_manager.cc:1347] Generated new cluster ID: f9b7d5c34b0640658073c334a0bd5e23
I20260620 20:19:02.321707 13276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:02.331668 13276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:02.332407 13276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:02.337536 13276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21: Generated new TSK 0
I20260620 20:19:02.338059 13276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:19:02.366323 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41099
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:02.466858 13289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:02.467077 13289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:02.467115 13289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:02.470952 13289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:02.470994 13289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:02.471081 13289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:19:02.475402 13289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:02.476275 13289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:02.477116 13289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:02.485428 13295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.485461 13294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.485582 13289 server_base.cc:1034] running on GCE node
W20260620 20:19:02.485596 13297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.485989 13289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:02.486574 13289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:02.487759 13289 hybrid_clock.cc:648] HybridClock initialized: now 1781986742487738 us; error 31 us; skew 500 ppm
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:02.490387 13289 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:19:02.491168 13289 webserver.cc:460] Webserver started at http://127.2.235.193:46751/ using document root <none> and password file <none>
I20260620 20:19:02.491712 13289 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:02.491794 13289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:02.492010 13289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:02.493811 13289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "3d493133f6aa4da581fb5cecd7273030"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "817fc00edf31a9cc0957a1a3ef12148d"
server_key_iv: "24ead4655fc9426226b64f24fa6f1878"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.494328 13289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "3d493133f6aa4da581fb5cecd7273030"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "817fc00edf31a9cc0957a1a3ef12148d"
server_key_iv: "24ead4655fc9426226b64f24fa6f1878"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.497665 13289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:19:02.500145 13304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.501204 13289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:19:02.501360 13289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "3d493133f6aa4da581fb5cecd7273030"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "817fc00edf31a9cc0957a1a3ef12148d"
server_key_iv: "24ead4655fc9426226b64f24fa6f1878"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.501451 13289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:02.517330 13289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:02.519891 13289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:02.520084 13289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:02.520622 13289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:02.521468 13289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:02.521538 13289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.521600 13289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:02.521648 13289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.530256 13289 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:41757
I20260620 20:19:02.530288 13417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:41757 every 8 connection(s)
I20260620 20:19:02.531108 13289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:02.531646 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 13289
I20260620 20:19:02.531769 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:19:02.532047 2991 external_mini_cluster.cc:1428] Setting key ab55ea24f51b83e6237d8b89c5383ea7
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:19:02.545571 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.532836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:33213 (local address 127.2.235.254:41099)
0620 20:19:02.532942 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.532948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.533916 (+ 968us) server_negotiation.cc:407] Connection header received
0620 20:19:02.535117 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.535119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.535184 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.535260 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.537040 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.537578 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.539381 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.539593 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.539664 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.542341 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.542344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.542346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.542366 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.543692 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.544161 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.544168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.544169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.544236 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.544581 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.544583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.544585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.544739 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.544836 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.545294 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.545406 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:19:02.546267 13420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.533299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41099 (local address 127.2.235.193:33213)
0620 20:19:02.533796 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.533832 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.534919 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.535475 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.535485 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.535948 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.536900 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.536913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.537700 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.537707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.539241 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.539258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.539751 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.539758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.540007 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.540550 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.540570 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.542190 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.543805 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.543810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.543822 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.544052 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.544319 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.544322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.544324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.544485 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.544836 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.544843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.545040 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.545239 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.545521 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260620 20:19:02.547474 13418 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41099
I20260620 20:19:02.547695 13418 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:02.548213 13418 heartbeater.cc:511] Master 127.2.235.254:41099 requested a full tablet report, sending...
I20260620 20:19:02.549875 13227 ts_manager.cc:194] Registered new tserver with Master: 3d493133f6aa4da581fb5cecd7273030 (127.2.235.193:41757)
I20260620 20:19:02.551440 13227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:33213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:19:02.559785 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41099
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:02.658581 13423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:02.658787 13423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:02.658825 13423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:02.662384 13423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:02.662429 13423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:02.662485 13423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:19:02.667232 13423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:02.668020 13423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:02.668968 13423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:02.677654 13428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.677707 13423 server_base.cc:1034] running on GCE node
W20260620 20:19:02.677668 13431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.677652 13429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.678218 13423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:02.678774 13423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:02.679929 13423 hybrid_clock.cc:648] HybridClock initialized: now 1781986742679911 us; error 29 us; skew 500 ppm
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:02.682636 13423 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:19:02.683351 13423 webserver.cc:460] Webserver started at http://127.2.235.194:33415/ using document root <none> and password file <none>
I20260620 20:19:02.683836 13423 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:02.683890 13423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:02.684095 13423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:02.685895 13423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "29978437164f4f66ae3e459076101abe"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "d665287b3b906bc347a21c5f259b7826"
server_key_iv: "82b9e2512fd284956ec34309676357ac"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.686414 13423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "29978437164f4f66ae3e459076101abe"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "d665287b3b906bc347a21c5f259b7826"
server_key_iv: "82b9e2512fd284956ec34309676357ac"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.689797 13423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:19:02.691851 13438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.692921 13423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:19:02.693056 13423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "29978437164f4f66ae3e459076101abe"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "d665287b3b906bc347a21c5f259b7826"
server_key_iv: "82b9e2512fd284956ec34309676357ac"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.693152 13423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:02.716688 13423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:02.719735 13423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:02.719933 13423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:02.720459 13423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:02.721191 13423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:02.721261 13423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.721369 13423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:02.721421 13423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.729702 13423 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:36531
I20260620 20:19:02.729751 13551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:36531 every 8 connection(s)
I20260620 20:19:02.730561 13423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:02.734998 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 13423
I20260620 20:19:02.735107 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:19:02.735407 2991 external_mini_cluster.cc:1428] Setting key fc4f025111ba41e96d8836750fb1520c
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:02.744629 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.732238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:32881 (local address 127.2.235.254:41099)
0620 20:19:02.732391 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.732396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.733145 (+ 749us) server_negotiation.cc:407] Connection header received
0620 20:19:02.734459 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.734462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.734504 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.734576 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.735922 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.736472 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.738090 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.738360 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.738506 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.740824 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.740829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.740833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.740863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.742593 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.743021 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.743024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.743026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.743107 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.743394 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.743398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.743400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.743683 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.743796 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.744221 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.744350 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:19:02.745182 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.732592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41099 (local address 127.2.235.194:32881)
0620 20:19:02.732989 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.733047 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.734261 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.734709 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.734715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.735172 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.735783 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.735807 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.736589 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.736593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.737952 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.737977 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.738531 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.738537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.738722 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.739250 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.739265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.740660 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.742704 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.742708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.742717 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.742927 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.743196 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.743198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.743199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.743305 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.743789 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.743794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.743989 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.744128 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.744335 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260620 20:19:02.746395 13552 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41099
I20260620 20:19:02.746637 13552 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:02.747188 13552 heartbeater.cc:511] Master 127.2.235.254:41099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:19:02.748308 13227 ts_manager.cc:194] Registered new tserver with Master: 29978437164f4f66ae3e459076101abe (127.2.235.194:36531)
I20260620 20:19:02.748888 13227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:32881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:19:02.761374 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41099
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:02.861023 13557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:02.861229 13557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:02.861349 13557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:02.865043 13557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:02.865087 13557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:02.865193 13557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:19:02.871059 13557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:02.871819 13557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:02.872596 13557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:02.880918 13563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.881022 13562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:02.880965 13565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:02.881222 13557 server_base.cc:1034] running on GCE node
I20260620 20:19:02.881646 13557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:02.882151 13557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:02.883296 13557 hybrid_clock.cc:648] HybridClock initialized: now 1781986742883269 us; error 37 us; skew 500 ppm
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:02.885694 13557 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:19:02.886381 13557 webserver.cc:460] Webserver started at http://127.2.235.195:44159/ using document root <none> and password file <none>
I20260620 20:19:02.886837 13557 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:02.886888 13557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:02.887104 13557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:19:02.888939 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "22326237f23745668a1179cd89f1bc5c"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "56416f18d561be7cc3030f28d1f08e6e"
server_key_iv: "1c12dcd6cdc612c95c849f87dc25c51b"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.889499 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "22326237f23745668a1179cd89f1bc5c"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "56416f18d561be7cc3030f28d1f08e6e"
server_key_iv: "1c12dcd6cdc612c95c849f87dc25c51b"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.892663 13557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:19:02.894807 13572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.895885 13557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 20:19:02.896019 13557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "22326237f23745668a1179cd89f1bc5c"
format_stamp: "Formatted at 2026-06-20 20:19:02 on dist-test-slave-8z53"
server_key: "56416f18d561be7cc3030f28d1f08e6e"
server_key_iv: "1c12dcd6cdc612c95c849f87dc25c51b"
server_key_version: "encryptionkey@0"
I20260620 20:19:02.896104 13557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:02.926367 13557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:02.928893 13557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:02.929085 13557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:02.929656 13557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:02.930441 13557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:02.930510 13557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.930573 13557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:02.930624 13557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:02.939221 13557 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:41213
I20260620 20:19:02.939252 13685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:41213 every 8 connection(s)
I20260620 20:19:02.939988 13557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:02.946238 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 13557
I20260620 20:19:02.946353 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:19:02.946662 2991 external_mini_cluster.cc:1428] Setting key 7c6b4532ff4b9456e9292502fbdaa444
Jun 20 20:19:02 dist-test-slave-8z53 krb5kdc[13182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986742, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:02.951860 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.942492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:43573 (local address 127.2.235.254:41099)
0620 20:19:02.942620 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.942623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.942635 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:19:02.943562 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.943564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.943602 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.943664 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.944653 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.945097 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.946074 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.946388 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.946511 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.948883 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.948887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.948890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.948912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.950094 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.950526 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.950531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.950533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.950603 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.950941 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.950944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.950945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.951098 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.951206 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.951613 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.951712 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:19:02.952499 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.941798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41099 (local address 127.2.235.195:43573)
0620 20:19:02.942173 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.942210 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.943395 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.943822 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.943828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.944121 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:02.944544 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.944555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.945183 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.945185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.945968 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.945976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.946530 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.946532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.946670 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.947166 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:02.947189 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:02.948738 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:02.950198 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.950203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.950214 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.950432 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.950685 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:02.950689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:02.950693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:02.950836 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:02.951205 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:02.951211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:02.951412 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.951575 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.951832 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20260620 20:19:02.953498 13686 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41099
I20260620 20:19:02.953677 13686 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:02.954031 13686 heartbeater.cc:511] Master 127.2.235.254:41099 requested a full tablet report, sending...
I20260620 20:19:02.954932 13227 ts_manager.cc:194] Registered new tserver with Master: 22326237f23745668a1179cd89f1bc5c (127.2.235.195:41213)
I20260620 20:19:02.955436 13227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:43573
I20260620 20:19:02.960100 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:19:02.968791 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.961776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.2.235.254:41099)
0620 20:19:02.961897 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.961900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.962088 (+ 188us) server_negotiation.cc:407] Connection header received
0620 20:19:02.962217 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.962222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.962259 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.962339 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:02.962998 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.963489 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.964335 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.964557 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.964654 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.965960 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:02.965963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:02.965966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:02.965995 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:02.967214 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.967613 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.967617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.967619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.967695 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:02.967957 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:02.967960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:02.967962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:02.968165 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:02.968305 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.968531 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.968636 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:19:02.971325 13227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:19:02.974089 13227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:19:02.986984 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.982173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:36531 (local address 127.0.0.1:45538)
0620 20:19:02.982674 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.982686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.982766 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.983333 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.983335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.983352 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:02.983427 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.983432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.984413 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.984416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.985481 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.985489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.986546 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.986550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.986636 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.986663 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.986710 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.986866 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":23,"thread_start_us":77,"threads_started":1}
I20260620 20:19:02.986996 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.981647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.2.235.193:41757)
0620 20:19:02.982222 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.982235 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.982261 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:19:02.982337 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.982344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.982521 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.982679 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:02.983147 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.984144 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.985692 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.986351 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.986469 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.986600 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.986755 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.986851 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":119,"threads_started":1}
I20260620 20:19:02.987294 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.981520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:41757 (local address 127.0.0.1:52222)
0620 20:19:02.981767 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.981805 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.981908 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.982864 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.982869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.982889 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:02.983014 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.983022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.984269 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.984272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.985563 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.985573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.986475 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.986479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.986581 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.986616 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.986719 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.987170 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":72,"thread_start_us":57,"threads_started":1}
I20260620 20:19:02.987438 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.981887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:41213 (local address 127.0.0.1:55898)
0620 20:19:02.982207 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:02.982221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:02.982302 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:02.983522 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:02.983525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:02.983540 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:02.983620 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.983625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.984845 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.984849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.985923 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:02.985931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.987052 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.987057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:02.987168 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.987188 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:19:02.987247 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 20:19:02.987306 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260620 20:19:02.987694 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.982512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.2.235.194:36531)
0620 20:19:02.982903 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.982914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.982936 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:19:02.983007 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.983012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.983197 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.983332 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:02.983522 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.984274 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.985602 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.986412 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.986865 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.987002 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.987449 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.987555 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
I20260620 20:19:02.987905 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:02.982131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.2.235.195:41213)
0620 20:19:02.982983 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:02.982994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:02.983011 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:19:02.983076 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:02.983079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:02.983286 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:02.983399 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:02.983733 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.984707 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.986035 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:02.986802 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:02.986888 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:02.986995 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:02.987687 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 20:19:02.987779 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":133,"threads_started":1}
I20260620 20:19:02.990187 13486 tablet_service.cc:1467] Processing CreateTablet for tablet fae29d36f2264daba759c2dd6c59d523 (DEFAULT_TABLE table=test-table [id=07c61cf680dc4fb38964808925e36d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:02.990278 13352 tablet_service.cc:1467] Processing CreateTablet for tablet fae29d36f2264daba759c2dd6c59d523 (DEFAULT_TABLE table=test-table [id=07c61cf680dc4fb38964808925e36d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:02.990521 13620 tablet_service.cc:1467] Processing CreateTablet for tablet fae29d36f2264daba759c2dd6c59d523 (DEFAULT_TABLE table=test-table [id=07c61cf680dc4fb38964808925e36d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:02.991158 13486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae29d36f2264daba759c2dd6c59d523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:02.991241 13352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae29d36f2264daba759c2dd6c59d523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:02.991279 13620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae29d36f2264daba759c2dd6c59d523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:02.997211 13703 tablet_bootstrap.cc:492] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Bootstrap starting.
I20260620 20:19:02.998209 13705 tablet_bootstrap.cc:492] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Bootstrap starting.
I20260620 20:19:02.998669 13704 tablet_bootstrap.cc:492] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: Bootstrap starting.
I20260620 20:19:03.000038 13703 tablet_bootstrap.cc:654] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:03.000521 13704 tablet_bootstrap.cc:654] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:03.000689 13705 tablet_bootstrap.cc:654] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:03.000970 13703 log.cc:824] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:03.001502 13705 log.cc:824] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:03.001730 13704 log.cc:824] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:03.002943 13703 tablet_bootstrap.cc:492] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: No bootstrap required, opened a new log
I20260620 20:19:03.003137 13703 ts_tablet_manager.cc:1397] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 20:19:03.003373 13705 tablet_bootstrap.cc:492] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: No bootstrap required, opened a new log
I20260620 20:19:03.003549 13704 tablet_bootstrap.cc:492] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: No bootstrap required, opened a new log
I20260620 20:19:03.003572 13705 ts_tablet_manager.cc:1397] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:19:03.003710 13704 ts_tablet_manager.cc:1397] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:19:03.006736 13705 raft_consensus.cc:348] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.006942 13705 raft_consensus.cc:374] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:03.007014 13705 raft_consensus.cc:729] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22326237f23745668a1179cd89f1bc5c, State: Initialized, Role: FOLLOWER
I20260620 20:19:03.006907 13703 raft_consensus.cc:348] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.007155 13703 raft_consensus.cc:374] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:03.007221 13703 raft_consensus.cc:729] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29978437164f4f66ae3e459076101abe, State: Initialized, Role: FOLLOWER
I20260620 20:19:03.007275 13704 raft_consensus.cc:348] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.007507 13704 raft_consensus.cc:374] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:03.007560 13704 raft_consensus.cc:729] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d493133f6aa4da581fb5cecd7273030, State: Initialized, Role: FOLLOWER
I20260620 20:19:03.007519 13705 consensus_queue.cc:260] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.007644 13703 consensus_queue.cc:260] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.007912 13704 consensus_queue.cc:260] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.008303 13552 heartbeater.cc:503] Master 127.2.235.254:41099 was elected leader, sending a full tablet report...
I20260620 20:19:03.008483 13705 ts_tablet_manager.cc:1428] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:19:03.008721 13704 ts_tablet_manager.cc:1428] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:19:03.008795 13686 heartbeater.cc:503] Master 127.2.235.254:41099 was elected leader, sending a full tablet report...
I20260620 20:19:03.009001 13418 heartbeater.cc:503] Master 127.2.235.254:41099 was elected leader, sending a full tablet report...
I20260620 20:19:03.009524 13703 ts_tablet_manager.cc:1428] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260620 20:19:03.032246 13419 tablet.cc:2369] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:03.129184 13710 raft_consensus.cc:482] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:03.129446 13710 raft_consensus.cc:504] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.130551 13710 leader_election.cc:288] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d493133f6aa4da581fb5cecd7273030 (127.2.235.193:41757), 22326237f23745668a1179cd89f1bc5c (127.2.235.195:41213)
I20260620 20:19:03.135360 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.130782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:40887 (local address 127.2.235.193:41757)
0620 20:19:03.130950 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.130964 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.130981 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:19:03.131128 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.131131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.131184 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.131262 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:03.131567 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.132326 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.134132 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.134724 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.134840 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.134891 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.135167 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.135241 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:19:03.135360 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.131078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:42759 (local address 127.2.235.195:41213)
0620 20:19:03.131214 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.131221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.131239 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:19:03.131295 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.131298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.131338 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.131422 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:03.131684 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.132364 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.134132 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.134724 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.134840 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.134891 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.135167 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.135241 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:19:03.135414 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.130730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:41757 (local address 127.2.235.194:40887)
0620 20:19:03.130897 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.130913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.130998 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.131284 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.131288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.131305 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:03.131462 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.131467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.132446 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.132449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.134019 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.134029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.134837 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.134840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.135009 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.135060 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.135134 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.135264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":200}
I20260620 20:19:03.135421 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.130753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:41213 (local address 127.2.235.194:42759)
0620 20:19:03.131108 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.131121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:03.131208 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:03.131461 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:03.131464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:03.131476 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:03.131584 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.131589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.132453 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.132473 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.134019 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:03.134029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.134837 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.134840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:03.135001 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.135063 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 20:19:03.135134 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:19:03.135264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":270,"thread_start_us":132,"threads_started":1}
I20260620 20:19:03.135996 13372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae29d36f2264daba759c2dd6c59d523" candidate_uuid: "29978437164f4f66ae3e459076101abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d493133f6aa4da581fb5cecd7273030" is_pre_election: true
I20260620 20:19:03.135998 13640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae29d36f2264daba759c2dd6c59d523" candidate_uuid: "29978437164f4f66ae3e459076101abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22326237f23745668a1179cd89f1bc5c" is_pre_election: true
I20260620 20:19:03.136276 13640 raft_consensus.cc:2454] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29978437164f4f66ae3e459076101abe in term 0.
I20260620 20:19:03.136279 13372 raft_consensus.cc:2454] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29978437164f4f66ae3e459076101abe in term 0.
I20260620 20:19:03.136698 13440 leader_election.cc:302] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22326237f23745668a1179cd89f1bc5c, 29978437164f4f66ae3e459076101abe; no voters:
I20260620 20:19:03.136938 13710 raft_consensus.cc:2781] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:03.137013 13710 raft_consensus.cc:482] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:03.137076 13710 raft_consensus.cc:3037] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:03.138039 13710 raft_consensus.cc:504] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.138341 13710 leader_election.cc:288] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [CANDIDATE]: Term 1 election: Requested vote from peers 3d493133f6aa4da581fb5cecd7273030 (127.2.235.193:41757), 22326237f23745668a1179cd89f1bc5c (127.2.235.195:41213)
I20260620 20:19:03.138653 13372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae29d36f2264daba759c2dd6c59d523" candidate_uuid: "29978437164f4f66ae3e459076101abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d493133f6aa4da581fb5cecd7273030"
I20260620 20:19:03.138676 13640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae29d36f2264daba759c2dd6c59d523" candidate_uuid: "29978437164f4f66ae3e459076101abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22326237f23745668a1179cd89f1bc5c"
I20260620 20:19:03.138777 13372 raft_consensus.cc:3037] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:03.138794 13640 raft_consensus.cc:3037] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:03.139714 13640 raft_consensus.cc:2454] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29978437164f4f66ae3e459076101abe in term 1.
I20260620 20:19:03.139721 13372 raft_consensus.cc:2454] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29978437164f4f66ae3e459076101abe in term 1.
I20260620 20:19:03.139941 13440 leader_election.cc:302] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22326237f23745668a1179cd89f1bc5c, 29978437164f4f66ae3e459076101abe; no voters:
I20260620 20:19:03.140127 13710 raft_consensus.cc:2781] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:03.140267 13710 raft_consensus.cc:686] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 1 LEADER]: Becoming Leader. State: Replica: 29978437164f4f66ae3e459076101abe, State: Running, Role: LEADER
I20260620 20:19:03.140563 13710 consensus_queue.cc:237] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } }
I20260620 20:19:03.143219 13225 catalog_manager.cc:5495] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe reported cstate change: term changed from 0 to 1, leader changed from <none> to 29978437164f4f66ae3e459076101abe (127.2.235.194). New cstate: current_term: 1 leader_uuid: "29978437164f4f66ae3e459076101abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29978437164f4f66ae3e459076101abe" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 36531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 } health_report { overall_health: UNKNOWN } } }
I20260620 20:19:03.173557 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.170106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.2.235.194:36531)
0620 20:19:03.170217 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:03.170222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:03.170235 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:19:03.170376 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:03.170379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:03.170428 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:03.170542 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:03.170808 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.171300 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.172150 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:03.172381 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:03.172486 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:03.172782 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:03.173150 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:03.173299 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:03.173359 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 20:19:03.173409 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 20:19:03.178622 13640 raft_consensus.cc:1261] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 1 FOLLOWER]: Refusing update from remote peer 29978437164f4f66ae3e459076101abe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:03.178622 13372 raft_consensus.cc:1261] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 1 FOLLOWER]: Refusing update from remote peer 29978437164f4f66ae3e459076101abe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:03.179318 13713 consensus_queue.cc:1035] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d493133f6aa4da581fb5cecd7273030" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:03.179467 13710 consensus_queue.cc:1035] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "22326237f23745668a1179cd89f1bc5c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:03.185973 13717 mvcc.cc:205] Tried to move back new op lower bound from 7299017700053229568 to 7299017699905982464. Current Snapshot: MvccSnapshot[applied={T|T < 7299017700053229568}]
I20260620 20:19:03.193953 13224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:03.194082 13224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:03.196061 13224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 848a95925d61456f913811e527116d21: Sending DeleteTablet for 3 replicas of tablet fae29d36f2264daba759c2dd6c59d523
I20260620 20:19:03.196755 13352 tablet_service.cc:1514] Processing DeleteTablet for tablet fae29d36f2264daba759c2dd6c59d523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:03 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:52222
I20260620 20:19:03.196779 13620 tablet_service.cc:1514] Processing DeleteTablet for tablet fae29d36f2264daba759c2dd6c59d523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:03 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:55898
I20260620 20:19:03.196812 13486 tablet_service.cc:1514] Processing DeleteTablet for tablet fae29d36f2264daba759c2dd6c59d523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:03 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:45538
I20260620 20:19:03.197145 13726 tablet_replica.cc:330] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: stopping tablet replica
I20260620 20:19:03.197175 13728 tablet_replica.cc:330] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: stopping tablet replica
I20260620 20:19:03.197429 13727 tablet_replica.cc:330] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030: stopping tablet replica
I20260620 20:19:03.197458 13726 raft_consensus.cc:2229] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:19:03.197712 13726 raft_consensus.cc:2258] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:03.197701 13727 raft_consensus.cc:2229] T fae29d36f2264daba759c2dd6c59d523 P 3d493133f6aa4da581fb5cecd7273030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:03.197767 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 13289
I20260620 20:19:03.200112 13728 raft_consensus.cc:2229] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:03.200404 13728 raft_consensus.cc:2258] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:03.200464 13726 ts_tablet_manager.cc:1905] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:03.201889 13728 ts_tablet_manager.cc:1905] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:03.203899 13726 ts_tablet_manager.cc:1918] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:03.204063 13726 log.cc:1196] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/fae29d36f2264daba759c2dd6c59d523
I20260620 20:19:03.204385 13728 ts_tablet_manager.cc:1918] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:03.204437 13726 ts_tablet_manager.cc:1939] T fae29d36f2264daba759c2dd6c59d523 P 29978437164f4f66ae3e459076101abe: Deleting consensus metadata
I20260620 20:19:03.204552 13728 log.cc:1196] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/fae29d36f2264daba759c2dd6c59d523
I20260620 20:19:03.204913 13728 ts_tablet_manager.cc:1939] T fae29d36f2264daba759c2dd6c59d523 P 22326237f23745668a1179cd89f1bc5c: Deleting consensus metadata
I20260620 20:19:03.205536 13211 catalog_manager.cc:4841] TS 29978437164f4f66ae3e459076101abe (127.2.235.194:36531): tablet fae29d36f2264daba759c2dd6c59d523 (table test-table [id=07c61cf680dc4fb38964808925e36d9a]) successfully deleted
I20260620 20:19:03.206084 13212 catalog_manager.cc:4841] TS 22326237f23745668a1179cd89f1bc5c (127.2.235.195:41213): tablet fae29d36f2264daba759c2dd6c59d523 (table test-table [id=07c61cf680dc4fb38964808925e36d9a]) successfully deleted
W20260620 20:19:03.206174 13211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:41757 (error 108)
I20260620 20:19:03.206290 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 13423
I20260620 20:19:03.208426 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:03.208178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:41757 (local address 127.0.0.1:52232)
0620 20:19:03.208298 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:03.208363 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:41757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:19:03.208566 13211 catalog_manager.cc:4584] TS 3d493133f6aa4da581fb5cecd7273030 (127.2.235.193:41757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fae29d36f2264daba759c2dd6c59d523: Network error: Client connection negotiation failed: client connection to 127.2.235.193:41757: connect: Connection refused (error 111)
I20260620 20:19:03.211784 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 13557
I20260620 20:19:03.216779 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 13195
2026-06-20T20:19:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2154 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:03 dist-test-slave-8z53 krb5kdc[13732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:03 dist-test-slave-8z53 krb5kdc[13732](info): set up 2 sockets
Jun 20 20:19:03 dist-test-slave-8z53 krb5kdc[13732](info): commencing operation
krb5kdc: starting...
W20260620 20:19:04.214267 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:19:04 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/rangeradmin_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/rangeradmin_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/rangerlookup_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/rangerlookup_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
I20260620 20:19:04.355679 2991 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9vOW7R/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/postgres -l logfile start
2026-06-20 20:19:06.407 UTC [13762] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:19:06.407 UTC [13762] LOG: listening on IPv4 address "127.2.235.212", port 42795
2026-06-20 20:19:06.410 UTC [13762] LOG: listening on Unix socket "/tmp/.s.PGSQL.42795"
2026-06-20 20:19:06.442 UTC [13765] LOG: database system was shut down at 2026-06-20 20:19:05 UTC
2026-06-20 20:19:06.446 UTC [13762] LOG: database system is ready to accept connections
I20260620 20:19:07.406322 2991 mini_postgres.cc:96] Postgres bound to 42795
2026-06-20 20:19:07.412 UTC [13773] FATAL: database "slave" does not exist
127.2.235.212:42795 - accepting connections
I20260620 20:19:07.413137 2991 mini_ranger.cc:161] Starting Ranger...
I20260620 20:19:07.427003 2991 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:19:07.800232 2991 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:19:07.800303 2991 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-admin
2026-06-20 20:19:08,062 [I] DB FLAVOR :POSTGRES
2026-06-20 20:19:08,063 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:19:08,063 [I] Checking connection..
2026-06-20 20:19:08,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:19:08,302 [I] Checking connection passed.
2026-06-20 20:19:08,302 [I] --------- Verifying version history table ---------
2026-06-20 20:19:08,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:19:08,534 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:19:08,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:19:08,752 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:19:08,753 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:19:08,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:19:08.979 UTC [13837] WARNING: there is no transaction in progress
2026-06-20 20:19:08,984 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:19:08,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:19:09,224 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:19:09,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:19:09,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:19:09,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:19:09,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8z53.c.gcp-upstream.internal','N') ;"
2026-06-20 20:19:09,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:19:10,127 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:19:10,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:19:10,334 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:19:10,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:19:10,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:19:10,774 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:19:10,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:19:11.578 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.592 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.606 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.618 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.632 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.649 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.703 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.708 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.720 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.728 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.737 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.749 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.756 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.762 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.767 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:11.773 UTC [13945] WARNING: there is no transaction in progress
2026-06-20 20:19:12,279 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:19:12,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:19:12,493 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:19:12,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:19:12,719 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:19:12,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:19:12,956 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:19:12,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:19:13,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:19:13,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8z53.c.gcp-upstream.internal';"
2026-06-20 20:19:13,645 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:19:13,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:19:13,873 [I] DB_PATCHES have already been applied
I20260620 20:19:13.880646 2991 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-admin:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9vOW7R/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:19:13.880723 2991 mini_ranger.cc:193] Using host: 127.2.235.212
I20260620 20:19:13.884443 2991 mini_ranger.cc:227] Ranger admin URL: http://127.2.235.212:40835
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:19:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:19:14.937616 2991 mini_ranger.cc:160] Time spent starting Ranger: real 7.525s user 0.005s sys 0.005s
Jun 20, 2026 8:19:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:19:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 8:19:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:19:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:19:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:19:33.930521 2991 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/rangerkms_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/rangerkms_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.235.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.235.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:19:34 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986774, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:19:34.020037 2991 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:19:34.030642 2991 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:19:34.727034 2991 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:19:34.727123 2991 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms
I20260620 20:19:34.727236 2991 mini_ranger_kms.cc:227] Using postgres at 127.2.235.212:42795
2026-06-20 20:19:34,899 [I] DB FLAVOR :POSTGRES
2026-06-20 20:19:34,899 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:19:34,899 [I] Checking connection
2026-06-20 20:19:34,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:19:35,115 [I] connection success
2026-06-20 20:19:35,115 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:19:35,115 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:19:35,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:19:35,328 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:19:35,329 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:19:35,329 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:19:35,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:42795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:19:35,558 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:19:35.984510 2991 mini_ranger_kms.cc:310] Created kms service
I20260620 20:19:36.144760 2991 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:19:36.183835 2991 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:19:36 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986774, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:36.564988 2991 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:19:36.565366 2991 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms:/tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:19:36.565415 2991 mini_ranger_kms.cc:240] Using host: 127.2.235.212
I20260620 20:19:36.569516 2991 mini_ranger_kms.cc:276] Ranger KMS PID: 14125
I20260620 20:19:36.569634 2991 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.235.212:59139
Jun 20, 2026 8:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:19:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:19:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:19:38.671610 2991 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.652s user 0.008s sys 0.149s
I20260620 20:19:38.671751 2991 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:19:38.671876 2991 mini_ranger_kms.cc:401] 127.2.235.212:59139/kms/v1/keys
Jun 20 20:19:38 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986778, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.235.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:19:39 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986778, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.235.212@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
Jun 20 20:19:39 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986778, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.235.212@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
Jun 20 20:19:41 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:19:41.773874 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40653
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:40653
--ranger_config_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:59139/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:41.897437 14176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:41.897639 14176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:41.897710 14176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:41.901227 14176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:19:41.901316 14176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:41.901337 14176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:41.901382 14176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:19:41.901412 14176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:19:41.906454 14176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.235.212:59139/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:40653
--ranger_config_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:41.907267 14176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:41.908087 14176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:41.916163 14181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:41.916163 14184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:41.916221 14182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:41.916227 14176 server_base.cc:1034] running on GCE node
I20260620 20:19:41.916903 14176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:41.917881 14176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:41.919062 14176 hybrid_clock.cc:648] HybridClock initialized: now 1781986781919037 us; error 35 us; skew 500 ppm
Jun 20 20:19:41 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986781, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:41.926651 14176 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:19:41.927372 14176 webserver.cc:460] Webserver started at http://127.2.235.254:38175/ using document root <none> and password file <none>
I20260620 20:19:41.927846 14176 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:41.927913 14176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:41.928131 14176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:19:41 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986781, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:41.978313 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "53407f3ed64643fc84ed5e4e86e8f4d7"
format_stamp: "Formatted at 2026-06-20 20:19:41 on dist-test-slave-8z53"
server_key: "dc4dbcd8132bf8f54c3430691ebeec83"
server_key_iv: "bcd5480cb48588466cd4da0f118e158b"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:41.979027 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "53407f3ed64643fc84ed5e4e86e8f4d7"
format_stamp: "Formatted at 2026-06-20 20:19:41 on dist-test-slave-8z53"
server_key: "dc4dbcd8132bf8f54c3430691ebeec83"
server_key_iv: "bcd5480cb48588466cd4da0f118e158b"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:41.984201 14176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:19:41.998255 14191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:41.999583 14176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 20:19:41.999709 14176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "53407f3ed64643fc84ed5e4e86e8f4d7"
format_stamp: "Formatted at 2026-06-20 20:19:41 on dist-test-slave-8z53"
server_key: "dc4dbcd8132bf8f54c3430691ebeec83"
server_key_iv: "bcd5480cb48588466cd4da0f118e158b"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:41.999814 14176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:42.018898 14176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:42.021682 14176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:42.021860 14176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:42.028926 14176 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:40653
I20260620 20:19:42.028946 14243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:40653 every 8 connection(s)
I20260620 20:19:42.029819 14176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:19:42.030673 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 14176
I20260620 20:19:42.030777 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:19:42.031039 2991 external_mini_cluster.cc:1428] Setting key f66796f23901d2df661e1a433494c6a9
I20260620 20:19:42.032923 14244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:42.038599 14244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7: Bootstrap starting.
Jun 20 20:19:42 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:42.041002 14244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:42.041878 14244 log.cc:824] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:42.043996 14244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7: No bootstrap required, opened a new log
I20260620 20:19:42.047116 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:42.032342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.2.235.254:40653)
0620 20:19:42.032776 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:42.032794 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:42.032845 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:19:42.033703 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:42.033725 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:42.034158 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:42.034489 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:42.035421 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:42.036878 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:42.038070 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:42.038730 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:42.038873 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:42.041382 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:42.041388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:42.041401 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:42.041427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:42.043855 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:42.044332 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:42.044340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:42.044348 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:42.044477 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:42.044735 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:42.044738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:42.044741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:42.045196 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:42.045424 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:42.045608 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:19:42.045872 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":133,"threads_started":1}
I20260620 20:19:42.048213 14244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40653 } }
I20260620 20:19:42.048408 14244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:42.048465 14244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53407f3ed64643fc84ed5e4e86e8f4d7, State: Initialized, Role: FOLLOWER
I20260620 20:19:42.048904 14244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40653 } }
I20260620 20:19:42.049113 14244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:19:42.049178 14244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:19:42.049292 14244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:42.050041 14244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40653 } }
I20260620 20:19:42.050362 14244 leader_election.cc:302] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53407f3ed64643fc84ed5e4e86e8f4d7; no voters:
I20260620 20:19:42.050625 14244 leader_election.cc:288] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:19:42.050757 14249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:42.050974 14249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [term 1 LEADER]: Becoming Leader. State: Replica: 53407f3ed64643fc84ed5e4e86e8f4d7, State: Running, Role: LEADER
I20260620 20:19:42.051280 14249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40653 } }
I20260620 20:19:42.051764 14244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:19:42.053059 14251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53407f3ed64643fc84ed5e4e86e8f4d7. Latest consensus state: current_term: 1 leader_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40653 } } }
I20260620 20:19:42.053208 14251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:42.053052 14250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53407f3ed64643fc84ed5e4e86e8f4d7" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40653 } } }
I20260620 20:19:42.053371 14250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7 [sys.catalog]: This master's current role is: LEADER
I20260620 20:19:42.053989 14258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:19:42.054234 14244 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:19:42.059162 14258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:19:42.064404 14258 catalog_manager.cc:1347] Generated new cluster ID: 5855a7fa9264446ab431de457bc64049
I20260620 20:19:42.064482 14258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:19:42.114354 14258 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:19:42.115623 14258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:19:42.121505 14258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7: Generated new TSK 0
I20260620 20:19:42.122287 14258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:19:42.711684 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:59139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40653
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:42.823117 14289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:42.823374 14289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:42.823448 14289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:42.827512 14289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:42.827584 14289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:42.827667 14289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:19:42.832656 14289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.235.212:59139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:42.833731 14289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:42.834730 14289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:42.843830 14297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:42.844079 14295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:42.844942 14294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:42.845723 14289 server_base.cc:1034] running on GCE node
I20260620 20:19:42.846171 14289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:42.846913 14289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:42.848140 14289 hybrid_clock.cc:648] HybridClock initialized: now 1781986782848123 us; error 36 us; skew 500 ppm
Jun 20 20:19:42 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:42.851634 14289 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:19:42.852639 14289 webserver.cc:460] Webserver started at http://127.2.235.193:40827/ using document root <none> and password file <none>
I20260620 20:19:42.853317 14289 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:42.853437 14289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:42.853796 14289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:19:42 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:42.872157 14289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "3e0bb2d77cfc469e82fe207dd4840035"
format_stamp: "Formatted at 2026-06-20 20:19:42 on dist-test-slave-8z53"
server_key: "218bad05ac0f1015a6600b4c00b63e53"
server_key_iv: "5b9ed079048a5e1b439019ccd29869dc"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:42.874552 14289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "3e0bb2d77cfc469e82fe207dd4840035"
format_stamp: "Formatted at 2026-06-20 20:19:42 on dist-test-slave-8z53"
server_key: "218bad05ac0f1015a6600b4c00b63e53"
server_key_iv: "5b9ed079048a5e1b439019ccd29869dc"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:42.879498 14289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:19:42.898291 14306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:42.899941 14289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:19:42.900070 14289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "3e0bb2d77cfc469e82fe207dd4840035"
format_stamp: "Formatted at 2026-06-20 20:19:42 on dist-test-slave-8z53"
server_key: "218bad05ac0f1015a6600b4c00b63e53"
server_key_iv: "5b9ed079048a5e1b439019ccd29869dc"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:42.900254 14289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:42.917743 14289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:42.921062 14289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:42.921305 14289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:42.921996 14289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:42.923015 14289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:42.923122 14289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:42.923233 14289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:42.923307 14289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:42.936607 14289 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40551
I20260620 20:19:42.937726 14289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:19:42.938205 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 14289
I20260620 20:19:42.938316 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:19:42.938637 2991 external_mini_cluster.cc:1428] Setting key 0ba1872f86253a3f8c4a21662a9c1479
I20260620 20:19:42.938328 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40551 every 8 connection(s)
Jun 20 20:19:42 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:42.958215 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:42.940733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40653 (local address 127.2.235.193:51631)
0620 20:19:42.942002 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:42.942054 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:42.943177 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:42.945835 (+ 2658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:42.945842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:42.946133 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:42.946697 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:42.946715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:42.947746 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:42.947749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:42.948959 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:42.948968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:42.949649 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:42.949653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:42.949832 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:42.951437 (+ 1605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:42.951455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:42.953139 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:42.955224 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:42.955229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:42.955241 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:42.955544 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:42.955881 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:42.955884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:42.955886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:42.956038 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:42.956473 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:42.956479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:42.956759 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 20:19:42.956962 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 20:19:42.957257 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":119,"threads_started":1}
I20260620 20:19:42.958894 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:42.940245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:51631 (local address 127.2.235.254:40653)
0620 20:19:42.945419 (+ 5174us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:42.945426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:42.945439 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:19:42.945473 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:42.945475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:42.945513 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:42.945582 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:42.946839 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:42.947604 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:42.949080 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:42.949410 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:42.949496 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:42.953332 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:42.953336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:42.953339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:42.953362 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:42.955110 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:42.955685 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:42.955689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:42.955691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:42.955774 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:42.956143 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:42.956146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:42.956148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:42.956343 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:42.956418 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:42.958599 (+ 2181us) server_negotiation.cc:299] Negotiation successful
0620 20:19:42.958738 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5107,"thread_start_us":79,"threads_started":1}
I20260620 20:19:42.960446 14421 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40653
I20260620 20:19:42.960662 14421 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:42.961149 14421 heartbeater.cc:511] Master 127.2.235.254:40653 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:19:42.963018 14208 ts_manager.cc:194] Registered new tserver with Master: 3e0bb2d77cfc469e82fe207dd4840035 (127.2.235.193:40551)
I20260620 20:19:42.964509 14208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:51631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:19:42.980181 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:59139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40653
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:19:43.103106 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:43.103335 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:43.103394 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:43.107525 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:43.107594 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:43.107689 14429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:19:43.113879 14429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.235.212:59139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:43.114946 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:43.116081 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:43.125073 14435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:43.125072 14436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:43.125396 14438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:43.125841 14429 server_base.cc:1034] running on GCE node
I20260620 20:19:43.126206 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:43.126858 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:43.128069 14429 hybrid_clock.cc:648] HybridClock initialized: now 1781986783128038 us; error 45 us; skew 500 ppm
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:43.130908 14429 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:19:43.131819 14429 webserver.cc:460] Webserver started at http://127.2.235.194:35257/ using document root <none> and password file <none>
I20260620 20:19:43.132336 14429 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:43.132422 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:43.132642 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:43.152264 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7"
format_stamp: "Formatted at 2026-06-20 20:19:43 on dist-test-slave-8z53"
server_key: "9bbe8bdd14861395e3d8246792a96e08"
server_key_iv: "54dd04bea63a74d1ba9811507fe699bb"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:43.152963 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7"
format_stamp: "Formatted at 2026-06-20 20:19:43 on dist-test-slave-8z53"
server_key: "9bbe8bdd14861395e3d8246792a96e08"
server_key_iv: "54dd04bea63a74d1ba9811507fe699bb"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:43.157624 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:19:43.171187 14445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:43.172459 14429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:19:43.172581 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7"
format_stamp: "Formatted at 2026-06-20 20:19:43 on dist-test-slave-8z53"
server_key: "9bbe8bdd14861395e3d8246792a96e08"
server_key_iv: "54dd04bea63a74d1ba9811507fe699bb"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:43.172708 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:43.184522 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:19:43.187274 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:43.187451 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:43.188043 14429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:43.188838 14429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:43.188898 14429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:43.188988 14429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:43.189038 14429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:43.198849 14429 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:44271
I20260620 20:19:43.198973 14558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:44271 every 8 connection(s)
I20260620 20:19:43.199644 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:19:43.206634 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 14429
I20260620 20:19:43.206758 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:19:43.207042 2991 external_mini_cluster.cc:1428] Setting key b194a1f73eac39bfc9f20e4db8834422
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:43.215984 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.202565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:57661 (local address 127.2.235.254:40653)
0620 20:19:43.202689 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.202693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.202712 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:19:43.203223 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.203226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.203299 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.203368 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:43.204558 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.205372 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.207968 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.208279 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.208350 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.212249 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:43.212252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:43.212255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:43.212279 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:43.213791 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:43.214379 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:43.214383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:43.214386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:43.214469 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:43.214826 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:43.214830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:43.214833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:43.215039 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:43.215130 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.215706 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.215822 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:19:43.217634 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.201598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40653 (local address 127.2.235.194:57661)
0620 20:19:43.202052 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.202101 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.203025 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.203515 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.203522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.203815 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:43.204388 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.204407 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.205525 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.205528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.207843 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.207853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.209426 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.209431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.209656 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.210272 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:43.210297 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:43.212093 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:43.213912 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:43.213921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:43.213938 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:43.214267 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:43.214568 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:43.214572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:43.214575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:43.214723 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:43.215208 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:43.215215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:43.215466 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.216400 (+ 934us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.216719 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":145,"threads_started":1}
I20260620 20:19:43.219003 14559 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40653
I20260620 20:19:43.219223 14559 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:43.219726 14559 heartbeater.cc:511] Master 127.2.235.254:40653 requested a full tablet report, sending...
I20260620 20:19:43.220775 14208 ts_manager.cc:194] Registered new tserver with Master: 1f69a6c849ad4a7cae0b6dc494fbeaa7 (127.2.235.194:44271)
I20260620 20:19:43.221474 14208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:57661
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:19:43.256145 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:59139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40653
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:19:43.440689 14564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:19:43.441336 14564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:19:43.441442 14564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:19:43.446952 14564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:19:43.447088 14564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:19:43.447208 14564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:19:43.454078 14564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.235.212:59139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:19:43.455413 14564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:19:43.456597 14564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:19:43.468837 14570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:19:43.468843 14569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:43.469565 14564 server_base.cc:1034] running on GCE node
W20260620 20:19:43.469527 14572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:19:43.470084 14564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:19:43.470746 14564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:19:43.471930 14564 hybrid_clock.cc:648] HybridClock initialized: now 1781986783471868 us; error 73 us; skew 500 ppm
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:19:43.475332 14564 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:19:43.476228 14564 webserver.cc:460] Webserver started at http://127.2.235.195:42541/ using document root <none> and password file <none>
I20260620 20:19:43.476814 14564 fs_manager.cc:359] Metadata directory not provided
I20260620 20:19:43.476889 14564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:19:43.477097 14564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:43.494148 14564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "9bf788a3d18d4d77b2024f7292be7a33"
format_stamp: "Formatted at 2026-06-20 20:19:43 on dist-test-slave-8z53"
server_key: "2e3c6e04e2091cc8cbf1f5e44b760545"
server_key_iv: "ef52a0ffbd7dfa4a0476a3a732ddf0be"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:43.494782 14564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "9bf788a3d18d4d77b2024f7292be7a33"
format_stamp: "Formatted at 2026-06-20 20:19:43 on dist-test-slave-8z53"
server_key: "2e3c6e04e2091cc8cbf1f5e44b760545"
server_key_iv: "ef52a0ffbd7dfa4a0476a3a732ddf0be"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:43.500106 14564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:19:43.524369 14579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:43.525929 14564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:19:43.526047 14564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "9bf788a3d18d4d77b2024f7292be7a33"
format_stamp: "Formatted at 2026-06-20 20:19:43 on dist-test-slave-8z53"
server_key: "2e3c6e04e2091cc8cbf1f5e44b760545"
server_key_iv: "ef52a0ffbd7dfa4a0476a3a732ddf0be"
server_key_version: "kuduclusterkey@0"
I20260620 20:19:43.526189 14564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:19:43.555011 14564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:43.557986 14564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:19:43.558151 14564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:19:43.558696 14564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:19:43.559598 14564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:19:43.559654 14564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:43.559710 14564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:19:43.559738 14564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:19:43.569445 14564 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:35115
I20260620 20:19:43.570523 14564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:19:43.571158 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:35115 every 8 connection(s)
I20260620 20:19:43.573901 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 14564
I20260620 20:19:43.574002 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:19:43.574323 2991 external_mini_cluster.cc:1428] Setting key 0416442ec82336e2e1dbdfce615c2f6f
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:19:43.589233 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.573353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:41529 (local address 127.2.235.254:40653)
0620 20:19:43.573467 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.573471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.576604 (+ 3133us) server_negotiation.cc:407] Connection header received
0620 20:19:43.577723 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.577726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.577771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.577814 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:43.579206 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.579970 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.581617 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.581981 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.582050 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.584738 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:43.584742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:43.584744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:43.584767 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:43.586292 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:43.587758 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:43.587762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:43.587763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:43.587836 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:43.588178 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:43.588181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:43.588182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:43.588358 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:43.588416 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.588956 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.589078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:19:43.590556 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.574016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40653 (local address 127.2.235.195:41529)
0620 20:19:43.576395 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.576518 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.577517 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.578169 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.578177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.578463 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:19:43.579029 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.579045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.580105 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.580108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.581500 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.581510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.582242 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.582245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.582431 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.583086 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:19:43.583103 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:19:43.584579 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:19:43.586410 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:43.586414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:43.586426 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:43.587130 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:43.587929 (+ 799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:19:43.587931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:19:43.587933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:19:43.588076 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:19:43.588483 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:19:43.588489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:19:43.588738 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.589440 (+ 702us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.589702 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":148,"threads_started":1}
I20260620 20:19:43.591627 14693 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40653
I20260620 20:19:43.591864 14693 heartbeater.cc:464] Registering TS with master...
I20260620 20:19:43.592386 14693 heartbeater.cc:511] Master 127.2.235.254:40653 requested a full tablet report, sending...
I20260620 20:19:43.593384 14208 ts_manager.cc:194] Registered new tserver with Master: 9bf788a3d18d4d77b2024f7292be7a33 (127.2.235.195:35115)
I20260620 20:19:43.594023 14208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:41529
I20260620 20:19:43.598966 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:19:43.620218 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.601614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.2.235.254:40653)
0620 20:19:43.601710 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.601714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.610674 (+ 8960us) server_negotiation.cc:407] Connection header received
0620 20:19:43.610847 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.610850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.610894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.610940 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:19:43.612201 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.612945 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.614868 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.615176 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.615258 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.616921 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:19:43.616924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:19:43.616927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:19:43.616950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:19:43.618493 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:43.618981 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:43.618985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:43.618986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:43.619053 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:19:43.619334 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:19:43.619337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:19:43.619338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:19:43.619521 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:19:43.619617 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.619953 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.620057 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:19:43.624370 14208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:19:43.628765 14208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:19:43.660378 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.639483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44271 (local address 127.0.0.1:39476)
0620 20:19:43.644172 (+ 4689us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.644213 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.644315 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.646737 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.646742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.646762 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:43.646871 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.646877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.649686 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.649689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.650993 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.651002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.659892 (+ 8890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.659895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.659999 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.660033 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.660138 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.660243 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4528,"thread_start_us":72,"threads_started":1}
I20260620 20:19:43.662431 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.642686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40551 (local address 127.0.0.1:40800)
0620 20:19:43.645792 (+ 3106us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.645803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.646980 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.647522 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.647525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.647539 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:43.647620 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.647625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.652408 (+ 4783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.652411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.655491 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.655518 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.662040 (+ 6522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.662044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.662153 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.662176 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.662244 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.662307 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3055,"thread_start_us":99,"threads_started":1}
I20260620 20:19:43.663072 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.640454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.2.235.194:44271)
0620 20:19:43.644843 (+ 4389us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.644856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.644880 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:19:43.644958 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.644963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.645189 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.646588 (+ 1399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:43.647726 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.648901 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.658817 (+ 9916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.659612 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.659709 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.659838 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.662820 (+ 2982us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.662924 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4278,"thread_start_us":87,"threads_started":1}
I20260620 20:19:43.664800 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.643059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.2.235.193:40551)
0620 20:19:43.645678 (+ 2619us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.645684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.647058 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 20:19:43.647138 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.647142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.647301 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.647411 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:43.651143 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.652282 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.660917 (+ 8635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.661741 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.661860 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.661982 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.664559 (+ 2577us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.664672 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2521,"thread_start_us":67,"threads_started":1}
I20260620 20:19:43.666061 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.641616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:35115 (local address 127.0.0.1:38182)
0620 20:19:43.644527 (+ 2911us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.644540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.644609 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.654482 (+ 9873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.654485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.654500 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:43.654587 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.654592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.656837 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.656840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.658665 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.658674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.665681 (+ 7007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.665686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.665791 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.665812 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.665894 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.665952 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2862,"thread_start_us":79,"threads_started":1}
I20260620 20:19:43.666525 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.641993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.2.235.195:35115)
0620 20:19:43.653972 (+ 11979us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.653981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.654000 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:19:43.654062 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.654066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.654240 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.654360 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:43.655645 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.656715 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.663481 (+ 6766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.664276 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.664373 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.664491 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.666273 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.666379 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11888,"thread_start_us":149,"threads_started":1}
I20260620 20:19:43.667063 14493 tablet_service.cc:1467] Processing CreateTablet for tablet e5eba5b22b554e679473ea05f3c5c314 (DEFAULT_TABLE table=test-table [id=54b51df9f9034600a63b4858dc849bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:43.667567 14354 tablet_service.cc:1467] Processing CreateTablet for tablet e5eba5b22b554e679473ea05f3c5c314 (DEFAULT_TABLE table=test-table [id=54b51df9f9034600a63b4858dc849bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:43.668154 14493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5eba5b22b554e679473ea05f3c5c314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:43.668557 14354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5eba5b22b554e679473ea05f3c5c314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:43.669430 14627 tablet_service.cc:1467] Processing CreateTablet for tablet e5eba5b22b554e679473ea05f3c5c314 (DEFAULT_TABLE table=test-table [id=54b51df9f9034600a63b4858dc849bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:19:43.670264 14627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5eba5b22b554e679473ea05f3c5c314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:19:43.676908 14712 tablet_bootstrap.cc:492] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: Bootstrap starting.
I20260620 20:19:43.679206 14712 tablet_bootstrap.cc:654] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:43.680572 14712 log.cc:824] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:43.681396 14713 tablet_bootstrap.cc:492] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Bootstrap starting.
I20260620 20:19:43.684011 14713 tablet_bootstrap.cc:654] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:43.684762 14713 log.cc:824] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:43.686668 14712 tablet_bootstrap.cc:492] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: No bootstrap required, opened a new log
I20260620 20:19:43.686846 14712 ts_tablet_manager.cc:1397] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 20:19:43.691147 14712 raft_consensus.cc:348] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.691421 14712 raft_consensus.cc:374] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:43.691475 14712 raft_consensus.cc:729] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0bb2d77cfc469e82fe207dd4840035, State: Initialized, Role: FOLLOWER
I20260620 20:19:43.691954 14712 consensus_queue.cc:260] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.692909 14712 ts_tablet_manager.cc:1428] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 20:19:43.693282 14421 heartbeater.cc:503] Master 127.2.235.254:40653 was elected leader, sending a full tablet report...
I20260620 20:19:43.699704 14713 tablet_bootstrap.cc:492] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: No bootstrap required, opened a new log
I20260620 20:19:43.699892 14713 ts_tablet_manager.cc:1397] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260620 20:19:43.703477 14715 tablet_bootstrap.cc:492] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Bootstrap starting.
I20260620 20:19:43.706035 14713 raft_consensus.cc:348] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.706358 14713 raft_consensus.cc:374] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:43.706411 14713 raft_consensus.cc:729] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf788a3d18d4d77b2024f7292be7a33, State: Initialized, Role: FOLLOWER
I20260620 20:19:43.707960 14715 tablet_bootstrap.cc:654] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Neither blocks nor log segments found. Creating new log.
I20260620 20:19:43.708827 14715 log.cc:824] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:19:43.709513 14713 consensus_queue.cc:260] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.710528 14713 ts_tablet_manager.cc:1428] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260620 20:19:43.710840 14693 heartbeater.cc:503] Master 127.2.235.254:40653 was elected leader, sending a full tablet report...
I20260620 20:19:43.713668 14715 tablet_bootstrap.cc:492] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: No bootstrap required, opened a new log
I20260620 20:19:43.713897 14715 ts_tablet_manager.cc:1397] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 20:19:43.717877 14715 raft_consensus.cc:348] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.718161 14715 raft_consensus.cc:374] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:19:43.718245 14715 raft_consensus.cc:729] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f69a6c849ad4a7cae0b6dc494fbeaa7, State: Initialized, Role: FOLLOWER
I20260620 20:19:43.718850 14715 consensus_queue.cc:260] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.719458 14559 heartbeater.cc:503] Master 127.2.235.254:40653 was elected leader, sending a full tablet report...
I20260620 20:19:43.719823 14715 ts_tablet_manager.cc:1428] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 20:19:43.822137 14694 tablet.cc:2369] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:19:43 dist-test-slave-8z53 krb5kdc[13732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:19:43.870343 14720 raft_consensus.cc:482] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:19:43.870555 14720 raft_consensus.cc:504] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.871820 14720 leader_election.cc:288] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf788a3d18d4d77b2024f7292be7a33 (127.2.235.195:35115), 3e0bb2d77cfc469e82fe207dd4840035 (127.2.235.193:40551)
I20260620 20:19:43.883589 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.872511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:47875 (local address 127.2.235.195:35115)
0620 20:19:43.872646 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.872653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.879432 (+ 6779us) server_negotiation.cc:407] Connection header received
0620 20:19:43.879653 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.879656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.879710 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.879758 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:43.880170 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.880833 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.882334 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.882913 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.883025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.883074 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.883388 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.883461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:19:43.884222 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.872152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:35115 (local address 127.2.235.194:47875)
0620 20:19:43.879352 (+ 7200us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.879367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.879512 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.879899 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.879902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.879922 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:43.880034 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.880041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.880988 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.880991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.882194 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.882208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.883163 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.883166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.883257 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.883279 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.883966 (+ 687us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.884055 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7117,"thread_start_us":116,"threads_started":1}
I20260620 20:19:43.884990 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eba5b22b554e679473ea05f3c5c314" candidate_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf788a3d18d4d77b2024f7292be7a33" is_pre_election: true
I20260620 20:19:43.885331 14629 raft_consensus.cc:2454] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f69a6c849ad4a7cae0b6dc494fbeaa7 in term 0.
I20260620 20:19:43.885979 14446 leader_election.cc:302] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f69a6c849ad4a7cae0b6dc494fbeaa7, 9bf788a3d18d4d77b2024f7292be7a33; no voters:
I20260620 20:19:43.886296 14720 raft_consensus.cc:2781] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:19:43.886368 14720 raft_consensus.cc:482] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:19:43.886409 14720 raft_consensus.cc:3037] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:43.887733 14720 raft_consensus.cc:504] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.888063 14720 leader_election.cc:288] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf788a3d18d4d77b2024f7292be7a33 (127.2.235.195:35115), 3e0bb2d77cfc469e82fe207dd4840035 (127.2.235.193:40551)
I20260620 20:19:43.888605 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eba5b22b554e679473ea05f3c5c314" candidate_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf788a3d18d4d77b2024f7292be7a33"
I20260620 20:19:43.888707 14629 raft_consensus.cc:3037] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:43.890563 14629 raft_consensus.cc:2454] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f69a6c849ad4a7cae0b6dc494fbeaa7 in term 1.
I20260620 20:19:43.890913 14446 leader_election.cc:302] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f69a6c849ad4a7cae0b6dc494fbeaa7, 9bf788a3d18d4d77b2024f7292be7a33; no voters:
I20260620 20:19:43.891098 14720 raft_consensus.cc:2781] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:19:43.891280 14720 raft_consensus.cc:686] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 1 LEADER]: Becoming Leader. State: Replica: 1f69a6c849ad4a7cae0b6dc494fbeaa7, State: Running, Role: LEADER
I20260620 20:19:43.891630 14720 consensus_queue.cc:237] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } }
I20260620 20:19:43.893591 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.873083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:56703 (local address 127.2.235.193:40551)
0620 20:19:43.873181 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.873185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.889482 (+ 16297us) server_negotiation.cc:407] Connection header received
0620 20:19:43.889660 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.889663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.889710 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.889754 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:19:43.890150 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.890858 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.892334 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.892922 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.893023 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.893078 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.893411 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.893483 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 20:19:43.894045 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.872729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40551 (local address 127.2.235.194:56703)
0620 20:19:43.889392 (+ 16663us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:43.889406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:19:43.889548 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:19:43.889872 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:19:43.889875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:19:43.889889 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:19:43.890001 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.890007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.891010 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.891013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.892193 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:19:43.892204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.893182 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.893185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:19:43.893289 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.893309 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:19:43.893863 (+ 554us) client_negotiation.cc:240] Negotiation successful
0620 20:19:43.893929 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16600,"thread_start_us":80,"threads_started":1}
I20260620 20:19:43.895041 14208 catalog_manager.cc:5495] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f69a6c849ad4a7cae0b6dc494fbeaa7 (127.2.235.194). New cstate: current_term: 1 leader_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 } health_report { overall_health: UNKNOWN } } }
I20260620 20:19:43.897927 14373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eba5b22b554e679473ea05f3c5c314" candidate_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0bb2d77cfc469e82fe207dd4840035" is_pre_election: true
I20260620 20:19:43.898193 14373 raft_consensus.cc:2454] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f69a6c849ad4a7cae0b6dc494fbeaa7 in term 0.
I20260620 20:19:43.898476 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5eba5b22b554e679473ea05f3c5c314" candidate_uuid: "1f69a6c849ad4a7cae0b6dc494fbeaa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0bb2d77cfc469e82fe207dd4840035"
I20260620 20:19:43.898566 14372 raft_consensus.cc:3037] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:19:43.899981 14372 raft_consensus.cc:2454] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f69a6c849ad4a7cae0b6dc494fbeaa7 in term 1.
W20260620 20:19:43.939635 14422 tablet.cc:2369] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:43.945581 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:43.941019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.2.235.194:44271)
0620 20:19:43.941130 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:19:43.941135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:19:43.941214 (+ 79us) server_negotiation.cc:407] Connection header received
0620 20:19:43.941375 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:19:43.941379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:19:43.941448 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:19:43.941571 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:19:43.941849 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.942643 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.943792 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:19:43.944078 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:19:43.944165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:19:43.944493 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:19:43.944952 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:19:43.945049 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:19:43.945419 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 20:19:43.945468 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 20:19:43.951536 14560 tablet.cc:2369] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:19:43.964854 14372 raft_consensus.cc:1261] T e5eba5b22b554e679473ea05f3c5c314 P 3e0bb2d77cfc469e82fe207dd4840035 [term 1 FOLLOWER]: Refusing update from remote peer 1f69a6c849ad4a7cae0b6dc494fbeaa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:43.965637 14724 consensus_queue.cc:1035] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0bb2d77cfc469e82fe207dd4840035" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:43.971747 14629 raft_consensus.cc:1261] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 1 FOLLOWER]: Refusing update from remote peer 1f69a6c849ad4a7cae0b6dc494fbeaa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:19:43.972384 14724 consensus_queue.cc:1035] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf788a3d18d4d77b2024f7292be7a33" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:19:43.991935 14731 mvcc.cc:205] Tried to move back new op lower bound from 7299017867112738816 to 7299017866823831552. Current Snapshot: MvccSnapshot[applied={T|T < 7299017867112738816}]
I20260620 20:19:43.992172 14729 mvcc.cc:205] Tried to move back new op lower bound from 7299017867112738816 to 7299017866823831552. Current Snapshot: MvccSnapshot[applied={T|T < 7299017867112738816}]
I20260620 20:19:44.002378 14208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:44.002537 14208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:19:44.005528 14208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53407f3ed64643fc84ed5e4e86e8f4d7: Sending DeleteTablet for 3 replicas of tablet e5eba5b22b554e679473ea05f3c5c314
I20260620 20:19:44.006435 14354 tablet_service.cc:1514] Processing DeleteTablet for tablet e5eba5b22b554e679473ea05f3c5c314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:44 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:40800
I20260620 20:19:44.007145 14493 tablet_service.cc:1514] Processing DeleteTablet for tablet e5eba5b22b554e679473ea05f3c5c314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:44 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:39476
I20260620 20:19:44.007552 14627 tablet_service.cc:1514] Processing DeleteTablet for tablet e5eba5b22b554e679473ea05f3c5c314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:19:44 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:38182
I20260620 20:19:44.008160 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 14289
I20260620 20:19:44.017444 14740 tablet_replica.cc:330] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: stopping tablet replica
I20260620 20:19:44.017803 14740 raft_consensus.cc:2229] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:19:44.018249 14740 raft_consensus.cc:2258] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:19:44.018618 14739 tablet_replica.cc:330] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: stopping tablet replica
I20260620 20:19:44.019529 14739 raft_consensus.cc:2229] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:19:44.020433 14740 ts_tablet_manager.cc:1905] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:44.020829 14739 raft_consensus.cc:2258] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 20:19:44.021770 14195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:40551 (error 108)
I20260620 20:19:44.022167 14739 ts_tablet_manager.cc:1905] T e5eba5b22b554e679473ea05f3c5c314 P 1f69a6c849ad4a7cae0b6dc494fbeaa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:19:44.022779 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 14429
I20260620 20:19:44.028479 14740 ts_tablet_manager.cc:1918] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:19:44.028637 14740 log.cc:1196] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/e5eba5b22b554e679473ea05f3c5c314
I20260620 20:19:44.028966 14740 ts_tablet_manager.cc:1939] T e5eba5b22b554e679473ea05f3c5c314 P 9bf788a3d18d4d77b2024f7292be7a33: Deleting consensus metadata
I20260620 20:19:44.030246 14192 catalog_manager.cc:4841] TS 9bf788a3d18d4d77b2024f7292be7a33 (127.2.235.195:35115): tablet e5eba5b22b554e679473ea05f3c5c314 (table test-table [id=54b51df9f9034600a63b4858dc849bbc]) successfully deleted
I20260620 20:19:44.033896 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:44.033660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40551 (local address 127.0.0.1:40810)
0620 20:19:44.033761 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:44.033826 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 20:19:44.034050 14195 catalog_manager.cc:4584] TS 3e0bb2d77cfc469e82fe207dd4840035 (127.2.235.193:40551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5eba5b22b554e679473ea05f3c5c314: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40551: connect: Connection refused (error 111)
I20260620 20:19:44.039458 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:19:44.039256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44271 (local address 127.0.0.1:39488)
0620 20:19:44.039370 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:19:44.039407 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.194:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:19:44.039914 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 14564
I20260620 20:19:44.051151 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 14176
I20260620 20:19:44.060326 2991 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:19:44.404145 2991 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:19:44.404248 2991 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:19:44.748487 2991 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:19:44.748 UTC [13762] LOG: received smart shutdown request
2026-06-20 20:19:44.751 UTC [13762] LOG: background worker "logical replication launcher" (PID 13771) exited with exit code 1
2026-06-20 20:19:44.751 UTC [13766] LOG: shutting down
2026-06-20 20:19:44.903 UTC [13762] LOG: database system is shut down
2026-06-20T20:19:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41797 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:19:45 dist-test-slave-8z53 krb5kdc[14750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:19:45 dist-test-slave-8z53 krb5kdc[14750](info): set up 2 sockets
Jun 20 20:19:45 dist-test-slave-8z53 krb5kdc[14750](info): commencing operation
krb5kdc: starting...
W20260620 20:19:45.977254 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:19:46 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:19:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/rangeradmin_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/rangeradmin_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/rangerlookup_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/rangerlookup_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
I20260620 20:19:46.115161 2991 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9vOW7R/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/postgres -l logfile start
2026-06-20 20:19:48.202 UTC [14780] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:19:48.202 UTC [14780] LOG: listening on IPv4 address "127.2.235.212", port 58625
2026-06-20 20:19:48.205 UTC [14780] LOG: listening on Unix socket "/tmp/.s.PGSQL.58625"
2026-06-20 20:19:48.224 UTC [14783] LOG: database system was shut down at 2026-06-20 20:19:47 UTC
2026-06-20 20:19:48.227 UTC [14780] LOG: database system is ready to accept connections
I20260620 20:19:49.145371 2991 mini_postgres.cc:96] Postgres bound to 58625
2026-06-20 20:19:49.151 UTC [14791] FATAL: database "slave" does not exist
127.2.235.212:58625 - accepting connections
I20260620 20:19:49.151710 2991 mini_ranger.cc:161] Starting Ranger...
I20260620 20:19:49.165236 2991 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:19:49.532157 2991 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:19:49.532243 2991 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-admin
2026-06-20 20:19:49,774 [I] DB FLAVOR :POSTGRES
2026-06-20 20:19:49,775 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:19:49,775 [I] Checking connection..
2026-06-20 20:19:49,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:19:50,017 [I] Checking connection passed.
2026-06-20 20:19:50,018 [I] --------- Verifying version history table ---------
2026-06-20 20:19:50,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:19:50,253 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:19:50,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:19:50,458 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:19:50,458 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:19:50,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:19:50.671 UTC [14855] WARNING: there is no transaction in progress
2026-06-20 20:19:50,677 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:19:50,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:19:50,881 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:19:50,882 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:19:50,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:19:51,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:19:51,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8z53.c.gcp-upstream.internal','N') ;"
2026-06-20 20:19:51,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:19:51,780 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:19:51,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:19:52,001 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:19:52,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:19:52,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:19:52,460 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:19:52,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:19:53.281 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.294 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.308 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.321 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.334 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.347 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.391 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.396 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.408 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.416 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.425 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.438 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.447 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.452 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.458 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53.464 UTC [14963] WARNING: there is no transaction in progress
2026-06-20 20:19:53,988 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:19:53,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:19:54,207 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:19:54,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:19:54,431 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:19:54,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:19:54,648 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:19:54,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:19:54,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:19:55,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8z53.c.gcp-upstream.internal';"
2026-06-20 20:19:55,282 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:19:55,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:19:55,515 [I] DB_PATCHES have already been applied
I20260620 20:19:55.520934 2991 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-admin:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9vOW7R/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:19:55.521013 2991 mini_ranger.cc:193] Using host: 127.2.235.212
I20260620 20:19:55.524334 2991 mini_ranger.cc:227] Ranger admin URL: http://127.2.235.212:60977
Jun 20, 2026 8:19:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:19:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:19:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:19:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:19:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:19:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:19:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:19:56.559459 2991 mini_ranger.cc:160] Time spent starting Ranger: real 7.408s user 0.000s sys 0.008s
Jun 20, 2026 8:20:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:20:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 8:20:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:20:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:20:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:20:16.060886 2991 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.235.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.235.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/rangerkms_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.235.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/rangerkms_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.235.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.235.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.235.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/HTTP_127.2.235.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:20:16 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986816, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:20:16.135448 2991 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:20:16.144286 2991 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:20:16.815183 2991 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:20:16.815266 2991 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms
I20260620 20:20:16.815287 2991 mini_ranger_kms.cc:227] Using postgres at 127.2.235.212:58625
2026-06-20 20:20:16,977 [I] DB FLAVOR :POSTGRES
2026-06-20 20:20:16,977 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:20:16,977 [I] Checking connection
2026-06-20 20:20:16,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:20:17,197 [I] connection success
2026-06-20 20:20:17,197 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:20:17,197 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:20:17,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:20:17,433 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:20:17,434 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:20:17,434 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:20:17,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.235.212:58625/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:20:17,672 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:20:18.084743 2991 mini_ranger_kms.cc:310] Created kms service
I20260620 20:20:18.271203 2991 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:20:18.311225 2991 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:20:18 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986816, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:18.667070 2991 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:20:18.667233 2991 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms:/tmp/dist-test-task9vOW7R/build/debug/bin/postgresql.jar:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9vOW7R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:20:18.667258 2991 mini_ranger_kms.cc:240] Using host: 127.2.235.212
I20260620 20:20:18.671520 2991 mini_ranger_kms.cc:276] Ranger KMS PID: 15143
I20260620 20:20:18.671618 2991 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.235.212:36229
Jun 20, 2026 8:20:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:20:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:20:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:20:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:20:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:20:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:20:20.784723 2991 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.649s user 0.014s sys 0.131s
I20260620 20:20:20.784852 2991 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:20:20.784945 2991 mini_ranger_kms.cc:401] 127.2.235.212:36229/kms/v1/keys
Jun 20 20:20:20 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.235.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:20:21 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.235.212@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
Jun 20 20:20:21 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.235.212@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
Jun 20 20:20:23 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:20:23.770022 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41483
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41483
--ranger_config_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:23.877403 15194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:23.877625 15194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:23.877662 15194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:23.881631 15194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:23.881680 15194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:23.881698 15194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:23.881740 15194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:23.881769 15194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:23.887125 15194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.235.212:36229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41483
--ranger_config_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:23.888046 15194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:23.889012 15194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:23.897209 15202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:23.897220 15200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:23.897320 15194 server_base.cc:1034] running on GCE node
W20260620 20:20:23.897266 15199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:23.898000 15194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:23.898953 15194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:23.900127 15194 hybrid_clock.cc:648] HybridClock initialized: now 1781986823900104 us; error 36 us; skew 500 ppm
Jun 20 20:20:23 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:23.905634 15194 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:23.906507 15194 webserver.cc:460] Webserver started at http://127.2.235.254:36993/ using document root <none> and password file <none>
I20260620 20:20:23.906960 15194 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:23.907009 15194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:23.907159 15194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:20:23 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:23.959208 15194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "92be51dc9809440a8019a6bab67ab498"
format_stamp: "Formatted at 2026-06-20 20:20:23 on dist-test-slave-8z53"
server_key: "56752e1b051b4a2eb27c57a3fa35a139"
server_key_iv: "61a53b74e451b2f41f74034cf717416b"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:23.960002 15194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "92be51dc9809440a8019a6bab67ab498"
format_stamp: "Formatted at 2026-06-20 20:20:23 on dist-test-slave-8z53"
server_key: "56752e1b051b4a2eb27c57a3fa35a139"
server_key_iv: "61a53b74e451b2f41f74034cf717416b"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:23.964093 15194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:20:23.977734 15209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:23.979075 15194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:23.979223 15194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "92be51dc9809440a8019a6bab67ab498"
format_stamp: "Formatted at 2026-06-20 20:20:23 on dist-test-slave-8z53"
server_key: "56752e1b051b4a2eb27c57a3fa35a139"
server_key_iv: "61a53b74e451b2f41f74034cf717416b"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:23.979359 15194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:23.991689 15194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:23.994403 15194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:23.994588 15194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:24.002319 15194 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:41483
I20260620 20:20:24.002324 15261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:41483 every 8 connection(s)
I20260620 20:20:24.003208 15194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:24.005971 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15194
I20260620 20:20:24.006095 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:20:24.006371 2991 external_mini_cluster.cc:1428] Setting key 7c5f04312f31600498567d89d01f8b13
I20260620 20:20:24.006356 15262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:24.013882 15262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Bootstrap starting.
Jun 20 20:20:24 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:24.016414 15262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:24.017669 15262 log.cc:824] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:24.019668 15262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: No bootstrap required, opened a new log
I20260620 20:20:24.019688 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:24.007697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.2.235.254:41483)
0620 20:20:24.008093 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:24.008106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:24.008140 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:20:24.008737 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:24.008751 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:24.009058 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:24.009423 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:24.010197 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:24.011119 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:24.011967 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:24.012405 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:24.012571 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:24.015005 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:24.015009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:24.015020 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:24.015044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:24.016737 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:24.017180 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:24.017185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:24.017190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:24.017345 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:24.017633 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:24.017636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:24.017638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:24.018089 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:24.018282 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:24.018612 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 20:20:24.018866 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260620 20:20:24.022970 15262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:24.023186 15262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:24.023250 15262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92be51dc9809440a8019a6bab67ab498, State: Initialized, Role: FOLLOWER
I20260620 20:20:24.023666 15262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:24.023814 15262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:24.023873 15262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:24.023957 15262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:24.024729 15262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:24.025043 15262 leader_election.cc:302] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92be51dc9809440a8019a6bab67ab498; no voters:
I20260620 20:20:24.025381 15262 leader_election.cc:288] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:24.025465 15267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:24.025665 15267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 LEADER]: Becoming Leader. State: Replica: 92be51dc9809440a8019a6bab67ab498, State: Running, Role: LEADER
I20260620 20:20:24.026000 15267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:24.026499 15262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:24.027196 15267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92be51dc9809440a8019a6bab67ab498. Latest consensus state: current_term: 1 leader_uuid: "92be51dc9809440a8019a6bab67ab498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } } }
I20260620 20:20:24.027299 15267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:24.028743 15262 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:20:24.028798 15268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92be51dc9809440a8019a6bab67ab498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } } }
I20260620 20:20:24.028975 15268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:24.029635 15275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:24.035173 15275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:24.041500 15275 catalog_manager.cc:1347] Generated new cluster ID: da4b2f40fcbf45a2b5d93108bc99182e
I20260620 20:20:24.041554 15275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:24.052292 15275 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:24.053573 15275 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:24.066229 15275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Generated new TSK 0
I20260620 20:20:24.067044 15275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:20:24.691495 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41483
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:24.834910 15307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:24.835165 15307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:24.835203 15307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:24.839219 15307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:24.839299 15307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:24.839402 15307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:20:24.845757 15307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.235.212:36229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:24.846699 15307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:24.847651 15307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:24.857012 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:24.857208 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:24.857496 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:24.857498 15307 server_base.cc:1034] running on GCE node
I20260620 20:20:24.857985 15307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:24.858675 15307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:24.859872 15307 hybrid_clock.cc:648] HybridClock initialized: now 1781986824859851 us; error 32 us; skew 500 ppm
Jun 20 20:20:24 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:24.863155 15307 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:20:24.864248 15307 webserver.cc:460] Webserver started at http://127.2.235.193:44509/ using document root <none> and password file <none>
I20260620 20:20:24.864924 15307 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:24.865037 15307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:24.865262 15307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:20:24 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:24.879321 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "8275436bd96146998a8a5ade746fe6c4"
format_stamp: "Formatted at 2026-06-20 20:20:24 on dist-test-slave-8z53"
server_key: "cf5b099f11ac9385852539c2b067d510"
server_key_iv: "517bfdd1dbea2ff5b487b5be4d053d29"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:24.879990 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "8275436bd96146998a8a5ade746fe6c4"
format_stamp: "Formatted at 2026-06-20 20:20:24 on dist-test-slave-8z53"
server_key: "cf5b099f11ac9385852539c2b067d510"
server_key_iv: "517bfdd1dbea2ff5b487b5be4d053d29"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:24.885036 15307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:20:24.905732 15324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:24.907308 15307 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260620 20:20:24.907485 15307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "8275436bd96146998a8a5ade746fe6c4"
format_stamp: "Formatted at 2026-06-20 20:20:24 on dist-test-slave-8z53"
server_key: "cf5b099f11ac9385852539c2b067d510"
server_key_iv: "517bfdd1dbea2ff5b487b5be4d053d29"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:24.907895 15307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:24.928807 15307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:24.932126 15307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:24.932327 15307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:24.933012 15307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:24.934010 15307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:24.934110 15307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:24.934231 15307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:24.934314 15307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:24.944113 15307 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40519
I20260620 20:20:24.945155 15307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:24.947886 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15307
I20260620 20:20:24.947995 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:24.948278 2991 external_mini_cluster.cc:1428] Setting key e57123b53b86b9afaf0f13e89a4dff3a
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:20:24.959514 15437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40519 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Jun 20 20:20:24 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:24.977947 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:24.965007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:38393 (local address 127.2.235.254:41483)
0620 20:20:24.965350 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:24.965363 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:24.965381 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:20:24.965449 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:24.965454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:24.965508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:24.965607 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:24.966961 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:24.967830 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:24.969512 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:24.969881 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:24.969985 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:24.972897 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:24.972901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:24.972904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:24.972929 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:24.974824 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:24.975501 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:24.975504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:24.975506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:24.975591 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:24.976444 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:24.976448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:24.976450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:24.976664 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:24.976756 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:24.977620 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 20:20:24.977738 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260620 20:20:24.979734 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:24.962755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41483 (local address 127.2.235.193:38393)
0620 20:20:24.963283 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:24.963334 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:24.964324 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:24.965935 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:24.965947 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:24.966289 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:24.966813 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:24.966827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:24.967975 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:24.967978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:24.969333 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:24.969344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:24.970151 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:24.970155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:24.970339 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:24.970998 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:24.971018 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:24.972717 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:24.975000 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:24.975006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:24.975020 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:24.975320 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:24.975748 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:24.975751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:24.975753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:24.976297 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:24.976883 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:24.976888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:24.977171 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 20:20:24.977466 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 20:20:24.978741 (+ 1275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260620 20:20:24.981442 15438 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41483
I20260620 20:20:24.981648 15438 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:24.982177 15438 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:24.983882 15226 ts_manager.cc:194] Registered new tserver with Master: 8275436bd96146998a8a5ade746fe6c4 (127.2.235.193:40519)
I20260620 20:20:24.985455 15226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:38393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:20:24.990408 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41483
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:20:25.135551 15446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:25.136010 15446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:25.136129 15446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:25.141892 15446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:25.142002 15446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:25.142115 15446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:20:25.149326 15446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.235.212:36229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:25.150614 15446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:25.151784 15446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:25.165225 15455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:25.165442 15453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:25.165526 15452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:25.166425 15446 server_base.cc:1034] running on GCE node
I20260620 20:20:25.166850 15446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:25.167510 15446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:25.168689 15446 hybrid_clock.cc:648] HybridClock initialized: now 1781986825168660 us; error 38 us; skew 500 ppm
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:25.171826 15446 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:20:25.172768 15446 webserver.cc:460] Webserver started at http://127.2.235.194:41777/ using document root <none> and password file <none>
I20260620 20:20:25.173442 15446 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:25.173574 15446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:25.173841 15446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:25.196461 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "a6299567a5ea445bb7e5725c3447e88f"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "0a50584a68804a491d4aa8f478f4efbf"
server_key_iv: "01004aed8f7d5991b7a3484606f68b06"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.197121 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "a6299567a5ea445bb7e5725c3447e88f"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "0a50584a68804a491d4aa8f478f4efbf"
server_key_iv: "01004aed8f7d5991b7a3484606f68b06"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.201352 15446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 20:20:25.214406 15462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.215757 15446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:25.215866 15446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "a6299567a5ea445bb7e5725c3447e88f"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "0a50584a68804a491d4aa8f478f4efbf"
server_key_iv: "01004aed8f7d5991b7a3484606f68b06"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.215974 15446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:25.235863 15446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:25.239008 15446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:25.239156 15446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:25.239699 15446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:25.240542 15446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:25.240597 15446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.240650 15446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:25.240677 15446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.249898 15446 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:40201
I20260620 20:20:25.250844 15446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:25.254842 15575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:40201 every 8 connection(s)
I20260620 20:20:25.257608 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15446
I20260620 20:20:25.257741 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:25.258073 2991 external_mini_cluster.cc:1428] Setting key 207a726042aa6063376082de52dec595
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:25.283401 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.257952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:58509 (local address 127.2.235.254:41483)
0620 20:20:25.258565 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:25.258570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:25.269706 (+ 11136us) server_negotiation.cc:407] Connection header received
0620 20:20:25.272399 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:25.272403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:25.272459 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:25.272561 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:25.273871 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.274644 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.276172 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.276498 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.276577 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:25.279265 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:25.279268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:25.279271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:25.279295 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:25.280940 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:25.281470 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:25.281473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:25.281475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:25.281544 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:25.281937 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:25.281941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:25.281943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:25.282279 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:25.282431 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:25.283062 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 20:20:25.283198 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260620 20:20:25.284236 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.256963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41483 (local address 127.2.235.194:58509)
0620 20:20:25.269567 (+ 12604us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:25.269617 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:25.271311 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:25.272738 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:25.272747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:25.273049 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:25.273708 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:25.273727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.274798 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.274802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:25.276035 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:25.276046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.276769 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.276772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:25.276960 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:25.277590 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:25.277610 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:25.279102 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:25.281060 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:25.281065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:25.281078 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:25.281364 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:25.281630 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:25.281633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:25.281635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:25.281799 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:25.282443 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:25.282449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:25.282771 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 20:20:25.282957 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 20:20:25.283252 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12393,"thread_start_us":126,"threads_started":1}
I20260620 20:20:25.285514 15576 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41483
I20260620 20:20:25.285761 15576 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:25.286283 15576 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:25.287518 15226 ts_manager.cc:194] Registered new tserver with Master: a6299567a5ea445bb7e5725c3447e88f (127.2.235.194:40201)
I20260620 20:20:25.288272 15226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:58509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:20:25.291705 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41483
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:25.446220 15581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:25.446564 15581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:25.446646 15581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:25.451866 15581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:25.451939 15581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:25.452023 15581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:20:25.458508 15581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.235.212:36229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:25.459637 15581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:25.460875 15581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:25.473090 15587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:25.473773 15586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:25.474077 15581 server_base.cc:1034] running on GCE node
W20260620 20:20:25.474308 15589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:25.474740 15581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:25.475452 15581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:25.476643 15581 hybrid_clock.cc:648] HybridClock initialized: now 1781986825476578 us; error 77 us; skew 500 ppm
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:25.480363 15581 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:20:25.481305 15581 webserver.cc:460] Webserver started at http://127.2.235.195:38269/ using document root <none> and password file <none>
I20260620 20:20:25.481920 15581 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:25.481992 15581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:25.482201 15581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:25.505774 15581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "94aec1878ab1470a9a25a433a49c5fb3"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "889e29a8007b3f41e1bd1256bb1e91a3"
server_key_iv: "3b82d6cdeb4a27a863860959b68686f1"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.506496 15581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "94aec1878ab1470a9a25a433a49c5fb3"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "889e29a8007b3f41e1bd1256bb1e91a3"
server_key_iv: "3b82d6cdeb4a27a863860959b68686f1"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.519826 15581 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260620 20:20:25.536875 15596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.538497 15581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:25.538632 15581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "94aec1878ab1470a9a25a433a49c5fb3"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "889e29a8007b3f41e1bd1256bb1e91a3"
server_key_iv: "3b82d6cdeb4a27a863860959b68686f1"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.538750 15581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:25.555990 15581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:25.559134 15581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:25.559299 15581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:25.559846 15581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:25.560786 15581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:25.560848 15581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.560906 15581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:25.560940 15581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.572230 15581 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:35271
I20260620 20:20:25.573215 15581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:25.577117 15709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:35271 every 8 connection(s)
I20260620 20:20:25.580281 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15581
I20260620 20:20:25.580400 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:25.580710 2991 external_mini_cluster.cc:1428] Setting key a2b403822a51156bcb97387c9134bb89
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:25.594846 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.581300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:46715 (local address 127.2.235.254:41483)
0620 20:20:25.581444 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:25.581449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:25.581870 (+ 421us) server_negotiation.cc:407] Connection header received
0620 20:20:25.583095 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:25.583098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:25.583151 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:25.583204 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:25.584799 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.585570 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.587239 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.587496 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.587565 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:25.590699 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:25.590704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:25.590707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:25.590735 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:25.592468 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:25.593048 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:25.593052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:25.593053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:25.593137 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:25.593574 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:25.593578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:25.593580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:25.593804 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:25.593881 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:25.594529 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 20:20:25.594657 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:20:25.596385 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:25.580232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41483 (local address 127.2.235.195:46715)
0620 20:20:25.581732 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:25.581783 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:25.582850 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:25.583699 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:25.583708 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:25.584037 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:25.584641 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:25.584658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.585753 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.585756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:25.587109 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:25.587120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:25.587738 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:25.587742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:25.587961 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:25.588748 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:25.588770 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:25.590496 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:25.592610 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:25.592615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:25.592628 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:25.592929 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:25.593267 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:25.593270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:25.593272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:25.593440 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:25.593967 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:25.593974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:25.594241 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 20:20:25.595081 (+ 840us) client_negotiation.cc:240] Negotiation successful
0620 20:20:25.595399 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":141,"threads_started":1}
I20260620 20:20:25.597537 15710 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41483
I20260620 20:20:25.597754 15710 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:25.598284 15710 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:25.599618 15226 ts_manager.cc:194] Registered new tserver with Master: 94aec1878ab1470a9a25a433a49c5fb3 (127.2.235.195:35271)
I20260620 20:20:25.600277 15226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:46715
I20260620 20:20:25.605515 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:25.606315 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15307
I20260620 20:20:25.613658 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15446
I20260620 20:20:25.619686 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15581
I20260620 20:20:25.627974 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15194
I20260620 20:20:25.635524 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41483
--webserver_interface=127.2.235.254
--webserver_port=36993
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41483
--ranger_config_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:25.741034 15713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:25.741307 15713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:25.741405 15713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:25.745419 15713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:25.745503 15713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:25.745524 15713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:25.745540 15713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:25.745575 15713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:25.750826 15713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41483
--ranger_config_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=36993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:25.751783 15713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:25.752733 15713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:25.760792 15718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:25.760910 15721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:25.760802 15719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:25.761302 15713 server_base.cc:1034] running on GCE node
I20260620 20:20:25.761862 15713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:25.762926 15713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:25.764117 15713 hybrid_clock.cc:648] HybridClock initialized: now 1781986825764103 us; error 28 us; skew 500 ppm
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:25.766924 15713 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:25.767673 15713 webserver.cc:460] Webserver started at http://127.2.235.254:36993/ using document root <none> and password file <none>
I20260620 20:20:25.768172 15713 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:25.768262 15713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:20:25 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986825, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:25.810264 15713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:25.812680 15729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:25.813992 15713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:25.814139 15713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "92be51dc9809440a8019a6bab67ab498"
format_stamp: "Formatted at 2026-06-20 20:20:23 on dist-test-slave-8z53"
server_key: "56752e1b051b4a2eb27c57a3fa35a139"
server_key_iv: "61a53b74e451b2f41f74034cf717416b"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:25.814697 15713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:25.831547 15713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:25.834486 15713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:25.834684 15713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:25.842193 15713 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:41483
I20260620 20:20:25.842198 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:41483 every 8 connection(s)
I20260620 20:20:25.843127 15713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:25.846941 15782 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:20:25.848491 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Bootstrap starting.
I20260620 20:20:25.852644 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15713
I20260620 20:20:25.853076 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:40519
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:41483
--webserver_port=44509
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:25.868012 15782 log.cc:824] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:25.875344 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:25.875813 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Bootstrap complete.
I20260620 20:20:25.881446 15782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:25.881904 15782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92be51dc9809440a8019a6bab67ab498, State: Initialized, Role: FOLLOWER
I20260620 20:20:25.882417 15782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:25.882550 15782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:25.882611 15782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:25.882705 15782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:20:25.884984 15782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:25.885368 15782 leader_election.cc:302] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92be51dc9809440a8019a6bab67ab498; no voters:
I20260620 20:20:25.885673 15782 leader_election.cc:288] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:20:25.885828 15786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:20:25.886132 15786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [term 2 LEADER]: Becoming Leader. State: Replica: 92be51dc9809440a8019a6bab67ab498, State: Running, Role: LEADER
I20260620 20:20:25.886485 15786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } }
I20260620 20:20:25.886829 15782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:25.888091 15787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "92be51dc9809440a8019a6bab67ab498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } } }
I20260620 20:20:25.888216 15787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:25.888911 15788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92be51dc9809440a8019a6bab67ab498. Latest consensus state: current_term: 2 leader_uuid: "92be51dc9809440a8019a6bab67ab498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92be51dc9809440a8019a6bab67ab498" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41483 } } }
I20260620 20:20:25.889005 15782 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:20:25.889104 15788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:25.889292 15796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:25.894758 15796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:25.895558 15796 catalog_manager.cc:1259] Loaded cluster ID: da4b2f40fcbf45a2b5d93108bc99182e
I20260620 20:20:25.895691 15796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:25.898121 15796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:25.898932 15796 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Loaded TSK: 0
I20260620 20:20:25.900166 15796 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:20:25.982371 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:25.982595 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:25.982673 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:25.987169 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:25.987262 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:25.987357 15784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:20:25.993556 15784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:40519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=44509
--tserver_master_addrs=127.2.235.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:25.994604 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:25.995487 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:26.004465 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:26.004465 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:26.005093 15784 server_base.cc:1034] running on GCE node
W20260620 20:20:26.005429 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:26.005925 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:26.006616 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:26.007834 15784 hybrid_clock.cc:648] HybridClock initialized: now 1781986826007804 us; error 53 us; skew 500 ppm
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:26.011160 15784 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:20:26.012179 15784 webserver.cc:460] Webserver started at http://127.2.235.193:44509/ using document root <none> and password file <none>
I20260620 20:20:26.012882 15784 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:26.013046 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:26.036258 15784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:26.050354 15830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.051573 15784 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260620 20:20:26.051694 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "8275436bd96146998a8a5ade746fe6c4"
format_stamp: "Formatted at 2026-06-20 20:20:24 on dist-test-slave-8z53"
server_key: "cf5b099f11ac9385852539c2b067d510"
server_key_iv: "517bfdd1dbea2ff5b487b5be4d053d29"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:26.052302 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:26.067850 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:26.070959 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:26.071151 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:26.071732 15784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:26.072690 15784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:26.072767 15784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.072831 15784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:26.072870 15784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.082909 15784 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40519
I20260620 20:20:26.084137 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:26.085193 15943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40519 every 8 connection(s)
I20260620 20:20:26.093945 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15784
I20260620 20:20:26.094409 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:40201
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:41483
--webserver_port=41777
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:26.109782 15946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.087696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41483 (local address 127.2.235.193:36191)
0620 20:20:26.089226 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:26.089303 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:26.090330 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:26.092161 (+ 1831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:26.092170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:26.092489 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:26.093109 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:26.093129 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.094943 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.094946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:26.096385 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:26.096396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.097477 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.097482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:26.097671 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.098310 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:26.098333 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:26.100061 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:26.106341 (+ 6280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:26.106348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:26.106365 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:26.106745 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:26.107181 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:26.107187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:26.107189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:26.107393 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:26.108098 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:26.108104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:26.108361 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 20:20:26.108552 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 20:20:26.108837 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"thread_start_us":125,"threads_started":1}
I20260620 20:20:26.109900 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.088885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:36191 (local address 127.2.235.254:41483)
0620 20:20:26.089489 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:26.089509 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:26.089568 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:20:26.090861 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:26.090886 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:26.091273 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:26.091627 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:26.093318 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.094788 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.096525 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.097298 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.097467 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.100259 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:26.100265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:26.100279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:26.100309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:26.106199 (+ 5890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.106903 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.106910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.106915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.107056 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.107510 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.107514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.107516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.107964 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:26.108089 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:26.108698 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 20:20:26.108978 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":112,"threads_started":1}
I20260620 20:20:26.112514 15944 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41483
I20260620 20:20:26.112764 15944 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.113325 15944 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
W20260620 20:20:26.115553 15944 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:41483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:20:26.115684 15944 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.115839 15944 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:26.116716 15944 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.116882 15944 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
W20260620 20:20:26.117693 15944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:20:26.229637 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:26.229935 15948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:26.230060 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:26.235103 15948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:26.235168 15948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:26.235248 15948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:20:26.241954 15948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:40201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=41777
--tserver_master_addrs=127.2.235.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:26.242761 15948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:26.243710 15948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:26.253060 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:26.253069 15953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:26.253073 15954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:26.253664 15948 server_base.cc:1034] running on GCE node
I20260620 20:20:26.254034 15948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:26.254683 15948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:26.255828 15948 hybrid_clock.cc:648] HybridClock initialized: now 1781986826255776 us; error 60 us; skew 500 ppm
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:26.258612 15948 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:20:26.259613 15948 webserver.cc:460] Webserver started at http://127.2.235.194:41777/ using document root <none> and password file <none>
I20260620 20:20:26.260257 15948 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:26.260377 15948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:26.290627 15948 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260620 20:20:26.300246 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.301584 15948 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260620 20:20:26.301724 15948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "a6299567a5ea445bb7e5725c3447e88f"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "0a50584a68804a491d4aa8f478f4efbf"
server_key_iv: "01004aed8f7d5991b7a3484606f68b06"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:26.302274 15948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:26.320513 15948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:26.324028 15948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:26.324249 15948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:26.324798 15948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:26.325846 15948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:26.325923 15948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.325994 15948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:26.326030 15948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.337899 15948 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:40201
I20260620 20:20:26.338117 16077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:40201 every 8 connection(s)
I20260620 20:20:26.339112 15948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:26.343643 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 15948
I20260620 20:20:26.344161 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.235.212:36229/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:35271
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:41483
--webserver_port=38269
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:26.354787 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.342481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:41163 (local address 127.2.235.254:41483)
0620 20:20:26.342626 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:26.342631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:26.342646 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:20:26.343219 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:26.343223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:26.343274 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:26.343324 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:26.344946 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.345776 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.347449 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.347778 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.347863 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.350790 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:26.350794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:26.350796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:26.350822 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:26.352462 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.353025 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.353029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.353031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.353119 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.353570 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.353573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.353575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.353779 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:26.353855 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:26.354508 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 20:20:26.354634 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:20:26.356410 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.341491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41483 (local address 127.2.235.194:41163)
0620 20:20:26.341996 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:26.342045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:26.342983 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:26.343791 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:26.343800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:26.344138 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:26.344777 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:26.344798 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.345938 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.345941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:26.347287 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:26.347301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.348056 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.348060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:26.348247 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.348898 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:26.348919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:26.350602 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:26.352588 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:26.352593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:26.352606 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:26.352906 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:26.353237 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:26.353240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:26.353242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:26.353448 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:26.353944 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:26.353950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:26.354217 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 20:20:26.355128 (+ 911us) client_negotiation.cc:240] Negotiation successful
0620 20:20:26.355421 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20260620 20:20:26.357950 16078 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41483
I20260620 20:20:26.358162 16078 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.358662 16078 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
W20260620 20:20:26.360139 16078 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:41483 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:20:26.360258 16078 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.360423 16078 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:26.361243 16078 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.361443 16078 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
W20260620 20:20:26.362237 16078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:20:26.478013 16081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:26.478318 16081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:26.478415 16081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:26.483112 16081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:26.483182 16081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:26.483251 16081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:20:26.488821 16081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.235.212:36229/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:35271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=38269
--tserver_master_addrs=127.2.235.254:41483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:26.489781 16081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:26.490669 16081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:26.500825 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:26.500872 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:26.501174 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:26.501705 16081 server_base.cc:1034] running on GCE node
I20260620 20:20:26.502104 16081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:26.502663 16081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:26.503829 16081 hybrid_clock.cc:648] HybridClock initialized: now 1781986826503805 us; error 33 us; skew 500 ppm
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:26.506783 16081 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:20:26.507658 16081 webserver.cc:460] Webserver started at http://127.2.235.195:38269/ using document root <none> and password file <none>
I20260620 20:20:26.508257 16081 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:26.508345 16081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:26.532634 16081 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 20:20:26.535568 16098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.537123 16081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:26.537228 16081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "94aec1878ab1470a9a25a433a49c5fb3"
format_stamp: "Formatted at 2026-06-20 20:20:25 on dist-test-slave-8z53"
server_key: "889e29a8007b3f41e1bd1256bb1e91a3"
server_key_iv: "3b82d6cdeb4a27a863860959b68686f1"
server_key_version: "kuduclusterkey@0"
I20260620 20:20:26.537853 16081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:26.572543 16081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:26.576000 16081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:26.576179 16081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:26.576794 16081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:26.577755 16081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:26.577823 16081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.577874 16081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:26.577913 16081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:26.588073 16081 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:35271
I20260620 20:20:26.588124 16211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:35271 every 8 connection(s)
I20260620 20:20:26.589066 16081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:26.593178 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16081
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:26.603567 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.594095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.2.235.254:41483)
0620 20:20:26.594326 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:26.594330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:26.594342 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:20:26.594389 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:26.594392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:26.594435 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:26.594532 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:26.595352 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.595842 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.597216 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.597494 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.597598 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.599961 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:26.599965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:26.599967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:26.599990 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:26.601764 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.602293 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.602297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.602299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.602396 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.602686 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.602689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.602691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.602913 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:26.603113 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:26.603278 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:20:26.603404 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260620 20:20:26.608810 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.591137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:36339 (local address 127.2.235.254:41483)
0620 20:20:26.591271 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:26.591275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:26.592089 (+ 814us) server_negotiation.cc:407] Connection header received
0620 20:20:26.592978 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:26.592982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:26.593038 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:26.593135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:26.594320 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.595078 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.596720 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.597064 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.597146 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.600874 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:26.600886 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:26.600890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:26.600938 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:26.605958 (+ 5020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.606554 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.606558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.606560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.606642 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:26.607052 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:26.607055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:26.607059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:26.607252 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:26.607345 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:26.608571 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0620 20:20:26.608652 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:20:26.610045 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:26.591498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41483 (local address 127.2.235.195:36339)
0620 20:20:26.591925 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:26.591971 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:26.592767 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:26.593333 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:26.593342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:26.593613 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:26.594199 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:26.594219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.595217 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.595221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:26.596574 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:26.596586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:26.597383 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:26.597387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:26.597561 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:26.598129 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:26.598195 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:26.600667 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:26.606071 (+ 5404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:26.606077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:26.606091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:26.606433 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:26.606726 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:26.606729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:26.606731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:26.606906 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:26.607367 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:26.607373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:26.608301 (+ 928us) client_negotiation.cc:769] Sending connection context
0620 20:20:26.608518 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:20:26.608819 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":137,"threads_started":1}
I20260620 20:20:26.611277 16212 heartbeater.cc:347] Connected to a master server at 127.2.235.254:41483
I20260620 20:20:26.611527 16212 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:26.611948 16212 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:26.613147 15746 ts_manager.cc:194] Registered new tserver with Master: 94aec1878ab1470a9a25a433a49c5fb3 (127.2.235.195:35271)
I20260620 20:20:26.614383 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:36339
Jun 20 20:20:26 dist-test-slave-8z53 krb5kdc[14750](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:27.117915 15944 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:27.118187 15944 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:27.119042 15746 ts_manager.cc:194] Registered new tserver with Master: 8275436bd96146998a8a5ade746fe6c4 (127.2.235.193:40519)
I20260620 20:20:27.119798 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:36191
Jun 20 20:20:27 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
I20260620 20:20:27.362475 16078 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:27.362727 16078 heartbeater.cc:511] Master 127.2.235.254:41483 requested a full tablet report, sending...
I20260620 20:20:27.363497 15746 ts_manager.cc:194] Registered new tserver with Master: a6299567a5ea445bb7e5725c3447e88f (127.2.235.194:40201)
I20260620 20:20:27.364240 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:41163
I20260620 20:20:27.364337 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:27.375365 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.366053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.2.235.254:41483)
0620 20:20:27.366582 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.366586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.366989 (+ 403us) server_negotiation.cc:407] Connection header received
0620 20:20:27.367121 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.367124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.367167 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.367250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:27.368061 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.368767 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.369923 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.370254 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.370355 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.371854 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:27.371858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:27.371869 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:27.371909 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:27.373751 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:27.374198 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:27.374201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:27.374203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:27.374292 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:27.374557 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:27.374560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:27.374561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:27.374752 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:27.374879 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.375103 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.375213 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":75,"threads_started":1}
I20260620 20:20:27.378955 15746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:27.383551 15746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:27.402376 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.394071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40519 (local address 127.0.0.1:43506)
0620 20:20:27.395494 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.395538 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:27.395672 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:27.396630 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:27.396637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:27.396660 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:27.396782 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.396787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.398802 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.398805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.400051 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.400061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.401936 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.401940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.402038 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.402061 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:20:27.402134 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:20:27.402244 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":10,"thread_start_us":85,"threads_started":1}
I20260620 20:20:27.402382 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.394789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.2.235.195:35271)
0620 20:20:27.395971 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.395977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.396943 (+ 966us) server_negotiation.cc:407] Connection header received
0620 20:20:27.397151 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.397155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.397340 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.397852 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:27.398218 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.399240 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.400818 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.401585 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.401654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.401786 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.402162 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.402249 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":69,"threads_started":1}
I20260620 20:20:27.402894 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.394113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:35271 (local address 127.0.0.1:60014)
0620 20:20:27.396816 (+ 2703us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.396829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:27.397018 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:27.398013 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:27.398016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:27.398031 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:27.398105 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.398109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.399406 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.399409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.400467 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.400476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.401882 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.401885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.401984 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.402015 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:20:27.402702 (+ 687us) client_negotiation.cc:240] Negotiation successful
0620 20:20:27.402771 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2642,"thread_start_us":89,"threads_started":1}
I20260620 20:20:27.403023 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.394186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.2.235.193:40519)
0620 20:20:27.396169 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.396175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.396197 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:20:27.396268 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.396274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.396466 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.396608 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:27.396930 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.398682 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.400197 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.401628 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.401740 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.401862 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.402793 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.402896 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":51,"threads_started":1}
I20260620 20:20:27.405807 16129 tablet_service.cc:1467] Processing CreateTablet for tablet 9c729bb955b0496e954859fa6b07c6c8 (DEFAULT_TABLE table=test-table [id=222267025e1d4f59bf29f15d50c79e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:27.405809 15876 tablet_service.cc:1467] Processing CreateTablet for tablet 9c729bb955b0496e954859fa6b07c6c8 (DEFAULT_TABLE table=test-table [id=222267025e1d4f59bf29f15d50c79e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:27.406855 15876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c729bb955b0496e954859fa6b07c6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:27.406857 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c729bb955b0496e954859fa6b07c6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:27.415321 16245 tablet_bootstrap.cc:492] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: Bootstrap starting.
I20260620 20:20:27.417830 16245 tablet_bootstrap.cc:654] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:27.418725 16245 log.cc:824] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:27.419387 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.395395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40201 (local address 127.0.0.1:54548)
0620 20:20:27.400654 (+ 5259us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.400666 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:27.400726 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:27.409391 (+ 8665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:27.409394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:27.409412 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:27.409486 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.409491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.416712 (+ 7221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.416716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.417910 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.417920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.419012 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.419016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.419118 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.419144 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:20:27.419190 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 20:20:27.419254 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5207,"thread_start_us":95,"threads_started":1}
I20260620 20:20:27.419488 16246 tablet_bootstrap.cc:492] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Bootstrap starting.
I20260620 20:20:27.420219 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.395793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.2.235.194:40201)
0620 20:20:27.403426 (+ 7633us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.403431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.403450 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:20:27.403507 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.403510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.403677 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.413235 (+ 9558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:27.413967 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.415031 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.418049 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.418755 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.419792 (+ 1037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.419943 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.419991 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.420074 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7546,"thread_start_us":111,"threads_started":1}
I20260620 20:20:27.421535 16246 tablet_bootstrap.cc:654] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:27.422295 16246 log.cc:824] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:27.422901 15996 tablet_service.cc:1467] Processing CreateTablet for tablet 9c729bb955b0496e954859fa6b07c6c8 (DEFAULT_TABLE table=test-table [id=222267025e1d4f59bf29f15d50c79e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:27.423839 15996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c729bb955b0496e954859fa6b07c6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:27.424196 16245 tablet_bootstrap.cc:492] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: No bootstrap required, opened a new log
I20260620 20:20:27.424371 16245 ts_tablet_manager.cc:1397] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 20:20:27.428418 16245 raft_consensus.cc:348] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.428752 16245 raft_consensus.cc:374] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:27.428850 16245 raft_consensus.cc:729] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8275436bd96146998a8a5ade746fe6c4, State: Initialized, Role: FOLLOWER
I20260620 20:20:27.429355 16245 consensus_queue.cc:260] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.430346 16245 ts_tablet_manager.cc:1428] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:20:27.430716 15944 heartbeater.cc:503] Master 127.2.235.254:41483 was elected leader, sending a full tablet report...
I20260620 20:20:27.431877 16250 tablet_bootstrap.cc:492] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Bootstrap starting.
I20260620 20:20:27.434224 16250 tablet_bootstrap.cc:654] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:27.435000 16250 log.cc:824] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:27.435078 16246 tablet_bootstrap.cc:492] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: No bootstrap required, opened a new log
I20260620 20:20:27.435343 16246 ts_tablet_manager.cc:1397] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260620 20:20:27.440009 16246 raft_consensus.cc:348] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.440254 16246 raft_consensus.cc:374] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:27.440351 16246 raft_consensus.cc:729] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94aec1878ab1470a9a25a433a49c5fb3, State: Initialized, Role: FOLLOWER
I20260620 20:20:27.440783 16246 consensus_queue.cc:260] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.441728 16246 ts_tablet_manager.cc:1428] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:20:27.443017 16212 heartbeater.cc:503] Master 127.2.235.254:41483 was elected leader, sending a full tablet report...
I20260620 20:20:27.443626 16250 tablet_bootstrap.cc:492] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: No bootstrap required, opened a new log
I20260620 20:20:27.443776 16250 ts_tablet_manager.cc:1397] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260620 20:20:27.447535 16250 raft_consensus.cc:348] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.447750 16250 raft_consensus.cc:374] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:27.447789 16250 raft_consensus.cc:729] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6299567a5ea445bb7e5725c3447e88f, State: Initialized, Role: FOLLOWER
I20260620 20:20:27.448196 16250 consensus_queue.cc:260] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.448750 16078 heartbeater.cc:503] Master 127.2.235.254:41483 was elected leader, sending a full tablet report...
I20260620 20:20:27.449007 16250 ts_tablet_manager.cc:1428] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
Jun 20 20:20:27 dist-test-slave-8z53 krb5kdc[14750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986826, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for HTTP/127.2.235.212@KRBTEST.COM
W20260620 20:20:27.590050 15945 tablet.cc:2369] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:27.591122 16213 tablet.cc:2369] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:27.591660 16079 tablet.cc:2369] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:27.643980 16253 raft_consensus.cc:482] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:27.644188 16253 raft_consensus.cc:504] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.645569 16253 leader_election.cc:288] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8275436bd96146998a8a5ade746fe6c4 (127.2.235.193:40519), 94aec1878ab1470a9a25a433a49c5fb3 (127.2.235.195:35271)
I20260620 20:20:27.651332 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.646866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:50991 (local address 127.2.235.193:40519)
0620 20:20:27.647004 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.647009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.647140 (+ 131us) server_negotiation.cc:407] Connection header received
0620 20:20:27.647322 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.647325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.647365 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.647418 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:27.647815 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.648514 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.650088 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.650683 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.650775 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.650824 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.651133 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.651203 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:20:27.651930 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.645985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40519 (local address 127.2.235.194:50991)
0620 20:20:27.647064 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.647078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:27.647212 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:27.647550 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:27.647552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:27.647569 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:27.647697 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.647703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.648648 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.648651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.649951 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.649961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.650909 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.650912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.651011 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.651033 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:20:27.651716 (+ 683us) client_negotiation.cc:240] Negotiation successful
0620 20:20:27.651778 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":102,"threads_started":1}
I20260620 20:20:27.652597 15895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c729bb955b0496e954859fa6b07c6c8" candidate_uuid: "a6299567a5ea445bb7e5725c3447e88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8275436bd96146998a8a5ade746fe6c4" is_pre_election: true
I20260620 20:20:27.652864 15895 raft_consensus.cc:2454] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6299567a5ea445bb7e5725c3447e88f in term 0.
I20260620 20:20:27.653354 15965 leader_election.cc:302] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8275436bd96146998a8a5ade746fe6c4, a6299567a5ea445bb7e5725c3447e88f; no voters:
I20260620 20:20:27.653649 16253 raft_consensus.cc:2781] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:27.653714 16253 raft_consensus.cc:482] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:27.653748 16253 raft_consensus.cc:3037] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:27.654652 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.646468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:40087 (local address 127.2.235.195:35271)
0620 20:20:27.646597 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.646604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.650969 (+ 4365us) server_negotiation.cc:407] Connection header received
0620 20:20:27.651135 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.651138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.651180 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.651233 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:27.651581 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.652237 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.653468 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.654049 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.654132 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.654173 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.654465 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.654530 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:20:27.655076 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.646400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:35271 (local address 127.2.235.194:40087)
0620 20:20:27.650901 (+ 4501us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.650913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:27.651025 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:27.651339 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:27.651343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:27.651361 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:27.651459 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.651465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.652353 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.652355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.653329 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:27.653337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.654257 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.654260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:27.654343 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.654360 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:20:27.654809 (+ 449us) client_negotiation.cc:240] Negotiation successful
0620 20:20:27.654885 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4428,"thread_start_us":94,"threads_started":1}
I20260620 20:20:27.655061 16253 raft_consensus.cc:504] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.655429 16253 leader_election.cc:288] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [CANDIDATE]: Term 1 election: Requested vote from peers 8275436bd96146998a8a5ade746fe6c4 (127.2.235.193:40519), 94aec1878ab1470a9a25a433a49c5fb3 (127.2.235.195:35271)
I20260620 20:20:27.655581 16166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c729bb955b0496e954859fa6b07c6c8" candidate_uuid: "a6299567a5ea445bb7e5725c3447e88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94aec1878ab1470a9a25a433a49c5fb3" is_pre_election: true
I20260620 20:20:27.655894 16166 raft_consensus.cc:2454] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6299567a5ea445bb7e5725c3447e88f in term 0.
I20260620 20:20:27.656140 15895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c729bb955b0496e954859fa6b07c6c8" candidate_uuid: "a6299567a5ea445bb7e5725c3447e88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8275436bd96146998a8a5ade746fe6c4"
I20260620 20:20:27.656247 15895 raft_consensus.cc:3037] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:27.656792 16166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c729bb955b0496e954859fa6b07c6c8" candidate_uuid: "a6299567a5ea445bb7e5725c3447e88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94aec1878ab1470a9a25a433a49c5fb3"
I20260620 20:20:27.656906 16166 raft_consensus.cc:3037] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:27.657521 15895 raft_consensus.cc:2454] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6299567a5ea445bb7e5725c3447e88f in term 1.
I20260620 20:20:27.657766 15965 leader_election.cc:302] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8275436bd96146998a8a5ade746fe6c4, a6299567a5ea445bb7e5725c3447e88f; no voters:
I20260620 20:20:27.657919 16253 raft_consensus.cc:2781] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:27.658084 16253 raft_consensus.cc:686] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 1 LEADER]: Becoming Leader. State: Replica: a6299567a5ea445bb7e5725c3447e88f, State: Running, Role: LEADER
I20260620 20:20:27.658138 16166 raft_consensus.cc:2454] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6299567a5ea445bb7e5725c3447e88f in term 1.
I20260620 20:20:27.658413 16253 consensus_queue.cc:237] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } }
I20260620 20:20:27.661268 15746 catalog_manager.cc:5495] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f reported cstate change: term changed from 0 to 1, leader changed from <none> to a6299567a5ea445bb7e5725c3447e88f (127.2.235.194). New cstate: current_term: 1 leader_uuid: "a6299567a5ea445bb7e5725c3447e88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6299567a5ea445bb7e5725c3447e88f" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40201 } health_report { overall_health: HEALTHY } } }
I20260620 20:20:27.687500 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.683716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.2.235.194:40201)
0620 20:20:27.683825 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:27.683829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:27.683838 (+ 9us) server_negotiation.cc:407] Connection header received
0620 20:20:27.683894 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:27.683897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:27.683941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:27.684045 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:27.684290 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.684744 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.685574 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:27.685771 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:27.685892 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:27.686244 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:27.686747 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:27.686866 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:27.687297 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 20:20:27.687352 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:20:27.693274 16166 raft_consensus.cc:1261] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 1 FOLLOWER]: Refusing update from remote peer a6299567a5ea445bb7e5725c3447e88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:27.693279 15895 raft_consensus.cc:1261] T 9c729bb955b0496e954859fa6b07c6c8 P 8275436bd96146998a8a5ade746fe6c4 [term 1 FOLLOWER]: Refusing update from remote peer a6299567a5ea445bb7e5725c3447e88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:27.693895 16258 consensus_queue.cc:1035] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8275436bd96146998a8a5ade746fe6c4" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:27.694061 16253 consensus_queue.cc:1035] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "94aec1878ab1470a9a25a433a49c5fb3" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:27.702266 16261 mvcc.cc:205] Tried to move back new op lower bound from 7299018046225088512 to 7299018046091399168. Current Snapshot: MvccSnapshot[applied={T|T < 7299018046225088512}]
I20260620 20:20:27.703748 16262 mvcc.cc:205] Tried to move back new op lower bound from 7299018046225088512 to 7299018046091399168. Current Snapshot: MvccSnapshot[applied={T|T < 7299018046225088512}]
I20260620 20:20:27.708719 16263 mvcc.cc:205] Tried to move back new op lower bound from 7299018046225088512 to 7299018046091399168. Current Snapshot: MvccSnapshot[applied={T|T < 7299018046225088512}]
I20260620 20:20:27.712966 15746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:27.713109 15746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:27.715623 15746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92be51dc9809440a8019a6bab67ab498: Sending DeleteTablet for 3 replicas of tablet 9c729bb955b0496e954859fa6b07c6c8
I20260620 20:20:27.716687 15996 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c729bb955b0496e954859fa6b07c6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:27 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:54548
I20260620 20:20:27.717412 16129 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c729bb955b0496e954859fa6b07c6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:27 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:60014
I20260620 20:20:27.717993 15876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c729bb955b0496e954859fa6b07c6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:27 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:43506
I20260620 20:20:27.719301 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15784
I20260620 20:20:27.727509 16271 tablet_replica.cc:330] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: stopping tablet replica
I20260620 20:20:27.727831 16271 raft_consensus.cc:2229] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:27.728101 16270 tablet_replica.cc:330] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: stopping tablet replica
I20260620 20:20:27.728440 16271 raft_consensus.cc:2258] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:27.728731 16270 raft_consensus.cc:2229] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:27.729303 16270 raft_consensus.cc:2258] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:27.730682 16271 ts_tablet_manager.cc:1905] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:27.733497 16271 ts_tablet_manager.cc:1918] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:27.734694 16270 ts_tablet_manager.cc:1905] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:27.737289 16270 ts_tablet_manager.cc:1918] T 9c729bb955b0496e954859fa6b07c6c8 P a6299567a5ea445bb7e5725c3447e88f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:27.737619 16271 log.cc:1196] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/9c729bb955b0496e954859fa6b07c6c8
W20260620 20:20:27.737885 15730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:40519 (error 108)
I20260620 20:20:27.738757 16271 ts_tablet_manager.cc:1939] T 9c729bb955b0496e954859fa6b07c6c8 P 94aec1878ab1470a9a25a433a49c5fb3: Deleting consensus metadata
I20260620 20:20:27.739907 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15948
I20260620 20:20:27.739955 15732 catalog_manager.cc:4841] TS 94aec1878ab1470a9a25a433a49c5fb3 (127.2.235.195:35271): tablet 9c729bb955b0496e954859fa6b07c6c8 (table test-table [id=222267025e1d4f59bf29f15d50c79e5a]) successfully deleted
I20260620 20:20:27.746698 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:27.746359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40519 (local address 127.0.0.1:43508)
0620 20:20:27.746542 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.746611 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 20:20:27.746950 15730 catalog_manager.cc:4584] TS 8275436bd96146998a8a5ade746fe6c4 (127.2.235.193:40519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c729bb955b0496e954859fa6b07c6c8: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40519: connect: Connection refused (error 111)
W20260620 20:20:27.748140 16243 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:20:27.747650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40201 (local address 127.0.0.1:54566)
0620 20:20:27.747791 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:27.747804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:27.747882 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:27.748063 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.194:40201: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 20:20:27.748697 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16081
I20260620 20:20:27.754769 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 15713
I20260620 20:20:27.759992 2991 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:20:28.103941 2991 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:20:28.104035 2991 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:20:28.447926 2991 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:20:28.448 UTC [14780] LOG: received smart shutdown request
2026-06-20 20:20:28.451 UTC [14780] LOG: background worker "logical replication launcher" (PID 14789) exited with exit code 1
2026-06-20 20:20:28.451 UTC [14784] LOG: shutting down
2026-06-20 20:20:28.583 UTC [14780] LOG: database system is shut down
2026-06-20T20:20:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (43683 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:28 dist-test-slave-8z53 krb5kdc[16284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:28 dist-test-slave-8z53 krb5kdc[16284](info): set up 2 sockets
Jun 20 20:20:28 dist-test-slave-8z53 krb5kdc[16284](info): commencing operation
krb5kdc: starting...
W20260620 20:20:29.697576 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:20:29 dist-test-slave-8z53 krb5kdc[16284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:20:29.802088 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43505
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43505,127.2.235.253:44627,127.2.235.252:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:29.902403 16297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:29.902602 16297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:29.902637 16297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:29.906230 16297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:29.906275 16297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:29.906294 16297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:29.906339 16297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:29.906446 16297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:29.911157 16297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43505,127.2.235.253:44627,127.2.235.252:37585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:29.912024 16297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:29.912849 16297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:29.920555 16303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:29.920564 16305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:29.920718 16297 server_base.cc:1034] running on GCE node
W20260620 20:20:29.920569 16302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:29.921427 16297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:29.922290 16297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:29.923447 16297 hybrid_clock.cc:648] HybridClock initialized: now 1781986829923422 us; error 37 us; skew 500 ppm
Jun 20 20:20:29 dist-test-slave-8z53 krb5kdc[16284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:29.930233 16297 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:29.931107 16297 webserver.cc:460] Webserver started at http://127.2.235.254:38329/ using document root <none> and password file <none>
I20260620 20:20:29.931602 16297 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:29.931658 16297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:29.931857 16297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:29.933732 16297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "d6d045cb665246c8bb706b4242911b77"
format_stamp: "Formatted at 2026-06-20 20:20:29 on dist-test-slave-8z53"
server_key: "da1502ffa49f9bb3785eec15b14b9c26"
server_key_iv: "8c92ed7fc24f3cff2305e99845c6471a"
server_key_version: "encryptionkey@0"
I20260620 20:20:29.934223 16297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "d6d045cb665246c8bb706b4242911b77"
format_stamp: "Formatted at 2026-06-20 20:20:29 on dist-test-slave-8z53"
server_key: "da1502ffa49f9bb3785eec15b14b9c26"
server_key_iv: "8c92ed7fc24f3cff2305e99845c6471a"
server_key_version: "encryptionkey@0"
I20260620 20:20:29.937630 16297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:20:29.939769 16312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:29.940891 16297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:29.941018 16297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "d6d045cb665246c8bb706b4242911b77"
format_stamp: "Formatted at 2026-06-20 20:20:29 on dist-test-slave-8z53"
server_key: "da1502ffa49f9bb3785eec15b14b9c26"
server_key_iv: "8c92ed7fc24f3cff2305e99845c6471a"
server_key_version: "encryptionkey@0"
I20260620 20:20:29.941119 16297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:29.959097 16297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:29.961509 16297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:29.961678 16297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:29.968214 16297 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:43505
I20260620 20:20:29.968240 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:43505 every 8 connection(s)
I20260620 20:20:29.969035 16297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:29.971434 16365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:29.974704 16365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:20:29.976946 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16297
I20260620 20:20:29.977082 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:20:29.977420 2991 external_mini_cluster.cc:1428] Setting key f03f28d58eb5b1995274c63f9b61b60c
Jun 20 20:20:29 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.253@KRBTEST.COM" created.
I20260620 20:20:29.988369 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.976757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.2.235.254:43505)
0620 20:20:29.977128 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:29.977148 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:29.977567 (+ 419us) server_negotiation.cc:407] Connection header received
0620 20:20:29.978520 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:29.978524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:29.978681 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:29.978851 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:29.979621 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:29.980433 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:29.981390 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:29.981727 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:29.981886 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:29.983920 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:29.983932 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:29.983940 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:29.983957 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:29.985443 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:29.985936 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:29.985943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:29.985948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:29.986057 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:29.986404 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:29.986410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:29.986412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:29.986732 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:29.986873 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:29.987203 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 20:20:29.987455 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":240,"thread_start_us":129,"threads_started":1}
I20260620 20:20:29.988373 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.976757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43505 (local address 127.0.0.1:53440)
0620 20:20:29.977430 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:29.977482 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:29.978224 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:29.978849 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:29.978863 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:29.979250 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:29.979464 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:29.979476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:29.980563 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:29.980566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:29.981233 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:29.981240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:29.981872 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:29.981876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:29.981979 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:29.982535 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:29.982549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:29.983789 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:29.985598 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:29.985605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:29.985613 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:29.985834 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:29.986185 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:29.986188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:29.986190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:29.986298 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:29.986850 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:29.986855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:29.986987 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 20:20:29.987145 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 20:20:29.987499 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":98,"thread_start_us":141,"threads_started":1}
I20260620 20:20:29.990552 16365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:20:29.991200 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.990973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52788)
0620 20:20:29.991073 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:29.991131 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:20:29.991382 16315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
I20260620 20:20:29.992122 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:29.991929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52792)
0620 20:20:29.992028 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:29.992075 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 20:20:29.992270 16365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:44627: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab.
Entry for principal kudu/127.2.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab.
I20260620 20:20:29.999611 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:44627
--webserver_interface=127.2.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab
--principal=kudu/127.2.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43505,127.2.235.253:44627,127.2.235.252:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:30.041498 16365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } attempt: 1
I20260620 20:20:30.042414 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.042093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52808)
0620 20:20:30.042249 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.042317 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:20:30.043149 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.042941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52812)
0620 20:20:30.043042 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.043092 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:20:30.043300 16365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:44627: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
W20260620 20:20:30.098738 16372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:30.098935 16372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:30.099004 16372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:30.102625 16372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:30.102669 16372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:30.102686 16372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:30.102701 16372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:30.102756 16372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:30.107689 16372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43505,127.2.235.253:44627,127.2.235.252:37585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab
--principal=kudu/127.2.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:44627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:30.108486 16372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:30.109335 16372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:20:30.111471 16365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } attempt: 2
I20260620 20:20:30.112232 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.111945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52828)
0620 20:20:30.112094 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.112156 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:20:30.112972 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.112726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52838)
0620 20:20:30.112840 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.112898 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:20:30.113166 16365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:44627: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44627: connect: Connection refused (error 111)
W20260620 20:20:30.116825 16380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:30.116844 16377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:30.117089 16372 server_base.cc:1034] running on GCE node
W20260620 20:20:30.117314 16378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:30.117794 16372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:30.118741 16372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:30.119956 16372 hybrid_clock.cc:648] HybridClock initialized: now 1781986830119918 us; error 50 us; skew 500 ppm
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:30.122444 16372 init.cc:377] Logged in from keytab as kudu/127.2.235.253@KRBTEST.COM (short username kudu)
I20260620 20:20:30.123220 16372 webserver.cc:460] Webserver started at http://127.2.235.253:39765/ using document root <none> and password file <none>
I20260620 20:20:30.123759 16372 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:30.123842 16372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:30.124058 16372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:30.125913 16372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data/instance:
uuid: "13a15bb59ee042208ec64fdf45a7fb05"
format_stamp: "Formatted at 2026-06-20 20:20:30 on dist-test-slave-8z53"
server_key: "5aa65fea50e70053b3e471e7e7ef80c1"
server_key_iv: "9dd07facabe88d0b80843280899eec3b"
server_key_version: "encryptionkey@0"
I20260620 20:20:30.126439 16372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal/instance:
uuid: "13a15bb59ee042208ec64fdf45a7fb05"
format_stamp: "Formatted at 2026-06-20 20:20:30 on dist-test-slave-8z53"
server_key: "5aa65fea50e70053b3e471e7e7ef80c1"
server_key_iv: "9dd07facabe88d0b80843280899eec3b"
server_key_version: "encryptionkey@0"
I20260620 20:20:30.129678 16372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:20:30.131747 16387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:30.132781 16372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:30.132916 16372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal
uuid: "13a15bb59ee042208ec64fdf45a7fb05"
format_stamp: "Formatted at 2026-06-20 20:20:30 on dist-test-slave-8z53"
server_key: "5aa65fea50e70053b3e471e7e7ef80c1"
server_key_iv: "9dd07facabe88d0b80843280899eec3b"
server_key_version: "encryptionkey@0"
I20260620 20:20:30.133019 16372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:30.143375 16372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:30.146245 16372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:30.146423 16372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:30.152518 16372 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.253:44627
I20260620 20:20:30.152542 16439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.253:44627 every 8 connection(s)
I20260620 20:20:30.153308 16372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
I20260620 20:20:30.154167 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16372
I20260620 20:20:30.154281 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-1/wal/instance
I20260620 20:20:30.154524 2991 external_mini_cluster.cc:1428] Setting key 708c75c07acd2a7999ce5bcdcdc5aaeb
I20260620 20:20:30.155714 16440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:30.158939 16440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.235.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.252@KRBTEST.COM" created.
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:30.172355 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.160450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.2.235.254:43505)
0620 20:20:30.160577 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.160581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.161572 (+ 991us) server_negotiation.cc:407] Connection header received
0620 20:20:30.162449 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.162452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.162505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.162586 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.163688 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.164299 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.165514 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.165744 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.165834 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.168549 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.168551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.168553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.168575 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.170462 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.171028 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.171030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.171031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.171095 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.171453 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.171455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.171456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.171598 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.171664 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.172113 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.172202 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:20:30.173025 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.160911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43505 (local address 127.0.0.1:53454)
0620 20:20:30.161425 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.161461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.162256 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.162772 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.162784 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.163076 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.163550 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.163563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.164433 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.164436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.165396 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.165409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.165891 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.165899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.166130 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.166686 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.166710 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.168393 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.170582 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.170589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.170604 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.170937 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.171180 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.171185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.171189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.171348 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.171682 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.171686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.171913 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.172050 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.172246 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260620 20:20:30.174396 16440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab.
Entry for principal kudu/127.2.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab.
I20260620 20:20:30.178356 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:37585
--webserver_interface=127.2.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab
--principal=kudu/127.2.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43505,127.2.235.253:44627,127.2.235.252:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:20:30.183743 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.174822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52842)
0620 20:20:30.174941 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.174957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.175040 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.175596 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.175601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.175774 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.175879 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.175884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.177014 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.177019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.177917 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.177927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.178821 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.178828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.178967 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.179337 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.179355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.180712 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.182436 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.182441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.182443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.182610 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.182866 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.182870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.182872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.182949 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.183319 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.183322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.183391 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.183498 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.183599 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:20:30.183807 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.174970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.2.235.253:44627)
0620 20:20:30.175242 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.175254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.175274 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:30.175313 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.175322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.175456 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.175553 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.175979 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.176795 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.178028 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.178560 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.178671 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.180845 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.180853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.180861 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.180879 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.182306 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.182712 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.182722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.182725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.182785 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.183029 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.183034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.183035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.183223 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.183347 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.183566 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.183667 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260620 20:20:30.185139 16440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:20:30.185762 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.185551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55232)
0620 20:20:30.185642 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.185705 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:20:30.186193 16388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
I20260620 20:20:30.187016 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.186826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55246)
0620 20:20:30.186928 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.186969 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:20:30.187265 16440 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:37585: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
I20260620 20:20:30.204321 16365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } attempt: 3
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:20:30.212581 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.204692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52858)
0620 20:20:30.204829 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.204855 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.204933 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.205149 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.205152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.205333 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.205455 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.205461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.206570 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.206574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.207439 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.207453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.207887 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.207907 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.208013 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.208424 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.208438 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.209670 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.211179 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.211184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.211186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.211370 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.211631 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.211635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.211637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.211717 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.212110 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.212113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.212207 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.212310 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.212430 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 20:20:30.212580 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.204789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.2.235.253:44627)
0620 20:20:30.204895 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.204898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.204910 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:20:30.205017 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.205019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.205050 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.205109 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.205566 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.206461 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.207557 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.207785 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.207854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.209802 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.209805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.209807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.209825 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.211075 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.211464 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.211470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.211472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.211543 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.211810 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.211814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.211815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.212020 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.212117 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.212357 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.212460 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 20:20:30.213212 16365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:20:30.213838 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.213653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55256)
0620 20:20:30.213749 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.213791 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 20:20:30.214466 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.214246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55258)
0620 20:20:30.214372 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.214415 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 20:20:30.214600 16365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:37585: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
I20260620 20:20:30.236485 16440 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } attempt: 1
I20260620 20:20:30.237025 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.236815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55262)
0620 20:20:30.236925 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.236974 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:20:30.237674 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.237479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55264)
0620 20:20:30.237579 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.237628 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:20:30.237794 16440 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:37585: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
I20260620 20:20:30.245743 16365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } attempt: 1
I20260620 20:20:30.246296 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.246085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55276)
0620 20:20:30.246192 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.246241 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:20:30.246959 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.246738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55278)
0620 20:20:30.246856 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.246909 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:20:30.247095 16365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:37585: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
W20260620 20:20:30.284914 16446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:30.285158 16446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:30.285280 16446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:30.289117 16446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:30.289193 16446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:30.289235 16446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:30.289295 16446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:30.289331 16446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:30.294186 16446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43505,127.2.235.253:44627,127.2.235.252:37585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab
--principal=kudu/127.2.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:30.295051 16446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:30.295926 16446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:30.303365 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:30.303467 16455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:30.303584 16453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:30.303714 16446 server_base.cc:1034] running on GCE node
I20260620 20:20:30.304193 16446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:30.305100 16446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:30.305934 16440 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } attempt: 2
I20260620 20:20:30.306254 16446 hybrid_clock.cc:648] HybridClock initialized: now 1781986830306231 us; error 46 us; skew 500 ppm
I20260620 20:20:30.306530 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.306280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55286)
0620 20:20:30.306415 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.306463 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:20:30.307189 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.306954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55300)
0620 20:20:30.307077 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.307129 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 20:20:30.307337 16440 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:37585: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:30.309163 16446 init.cc:377] Logged in from keytab as kudu/127.2.235.252@KRBTEST.COM (short username kudu)
I20260620 20:20:30.310032 16446 webserver.cc:460] Webserver started at http://127.2.235.252:33463/ using document root <none> and password file <none>
I20260620 20:20:30.310550 16446 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:30.310642 16446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:30.310847 16446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:30.312687 16446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data/instance:
uuid: "5a17155ac90b49e3b5ee448918231248"
format_stamp: "Formatted at 2026-06-20 20:20:30 on dist-test-slave-8z53"
server_key: "a7e4d22802dc4773b83467997762d7f6"
server_key_iv: "ebf4131eacaf754379afec93f1a87d2a"
server_key_version: "encryptionkey@0"
I20260620 20:20:30.313295 16446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal/instance:
uuid: "5a17155ac90b49e3b5ee448918231248"
format_stamp: "Formatted at 2026-06-20 20:20:30 on dist-test-slave-8z53"
server_key: "a7e4d22802dc4773b83467997762d7f6"
server_key_iv: "ebf4131eacaf754379afec93f1a87d2a"
server_key_version: "encryptionkey@0"
I20260620 20:20:30.316758 16446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:30.319259 16462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:30.320317 16446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:30.320461 16446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal
uuid: "5a17155ac90b49e3b5ee448918231248"
format_stamp: "Formatted at 2026-06-20 20:20:30 on dist-test-slave-8z53"
server_key: "a7e4d22802dc4773b83467997762d7f6"
server_key_iv: "ebf4131eacaf754379afec93f1a87d2a"
server_key_version: "encryptionkey@0"
I20260620 20:20:30.320571 16446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:30.322243 16365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } attempt: 2
I20260620 20:20:30.322927 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.322658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55312)
0620 20:20:30.322799 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.322847 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 20:20:30.323621 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.323419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55318)
0620 20:20:30.323524 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.323573 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:20:30.323769 16365 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:37585: Network error: Client connection negotiation failed: client connection to 127.2.235.252:37585: connect: Connection refused (error 111)
I20260620 20:20:30.351861 16446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:30.354913 16446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:30.355108 16446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:30.361828 16446 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.252:37585
I20260620 20:20:30.361836 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.252:37585 every 8 connection(s)
I20260620 20:20:30.362696 16446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
I20260620 20:20:30.363976 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16446
I20260620 20:20:30.364101 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986686246774-2991-0/minicluster-data/master-2/wal/instance
I20260620 20:20:30.364395 2991 external_mini_cluster.cc:1428] Setting key 8dcef80228f66d59921e4db35d48fddc
I20260620 20:20:30.365665 16515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:30.369457 16515 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:30.378428 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.365594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.2.235.252:37585)
0620 20:20:30.366002 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.366017 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.366053 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:20:30.366668 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.366682 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.366973 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.367246 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.368093 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.369029 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.370143 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.370583 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.370749 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.373703 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.373708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.373716 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.373742 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.375639 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.376069 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.376075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.376081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.376256 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.376647 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.376651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.376652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.377053 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.377329 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.377544 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.377823 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260620 20:20:30.379803 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.371036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43505 (local address 127.0.0.1:53464)
0620 20:20:30.371470 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.371519 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.371644 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.371961 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.371967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.372419 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.372560 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.372566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.373588 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.373593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.374218 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.374226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.374652 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.374656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.374764 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.375201 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.375211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.376604 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.378364 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.378372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.378377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.378621 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.378891 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.378894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.378895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.378974 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.379388 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.379392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.379476 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.379581 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.379693 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":135,"threads_started":1}
I20260620 20:20:30.379925 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.371144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.2.235.254:43505)
0620 20:20:30.371284 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.371288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.371616 (+ 328us) server_negotiation.cc:407] Connection header received
0620 20:20:30.371755 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.371757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.371802 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.371883 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.372694 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.373467 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.374335 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.374549 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.374653 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.376742 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.376745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.376747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.376769 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.378249 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.378705 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.378710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.378712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.378788 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.379059 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.379064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.379066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.379278 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.379380 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.379652 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.379757 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:20:30.380999 16515 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:30.388720 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.381445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44627 (local address 127.0.0.1:52872)
0620 20:20:30.381574 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.381587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.381656 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.381904 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.381907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.382070 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.382145 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.382150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.382833 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.382839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.383343 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.383350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.383774 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.383777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.383865 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.384240 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.384250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.385339 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.387224 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.387227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.387229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.387419 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.387699 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.387702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.387703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.387775 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.388240 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.388243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.388330 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.388451 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.388577 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:20:30.388785 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.381590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.2.235.253:44627)
0620 20:20:30.381706 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.381710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.381729 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:20:30.381763 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.381765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.381798 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.381889 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.382261 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.382733 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.383452 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.383675 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.383774 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.385486 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.385489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.385491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.385514 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.387110 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.387511 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.387515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.387517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.387598 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.387867 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.387870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.387872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.388088 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.388190 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.388503 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.388626 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:20:30.389086 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.380862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.2.235.254:43505)
0620 20:20:30.380955 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.380961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.380974 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:20:30.381079 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.381082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.381124 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.381213 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.381857 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.382321 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.383077 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.383318 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.383425 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.385990 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.385994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.385997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.386019 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.387516 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.387908 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.387910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.387911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.387975 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.388253 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.388257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.388258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.388479 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.388680 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.388820 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.388947 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:20:30.389698 16515 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:20:30.397753 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.390063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55334)
0620 20:20:30.390149 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.390161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.390229 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.390471 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.390477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.390630 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.390689 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.390693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.391798 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.391801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.392298 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.392308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.392742 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.392746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.392813 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.393228 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.393238 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.394518 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.396421 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.396424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.396428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.396663 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.396909 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.396913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.396915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.396999 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.397350 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.397353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.397444 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.397563 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.397663 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 20:20:30.398233 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.390167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.2.235.252:37585)
0620 20:20:30.390267 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.390270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.390281 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:20:30.390312 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.390318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.390359 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.390441 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.390792 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.391692 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.392411 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.392630 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.392726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.394646 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.394650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.394653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.394675 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.396307 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.396759 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.396761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.396762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.396824 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.397083 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.397085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.397086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.397219 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.397358 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.398004 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.398097 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 20:20:30.398280 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.390104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.2.235.253:44627)
0620 20:20:30.390202 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.390207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.390253 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:20:30.390356 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.390363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.390399 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.390488 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.391147 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.391580 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.392328 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.392580 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.392647 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.395168 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.395171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.395174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.395202 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.396732 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.397143 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.397148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.397149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.397224 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.397540 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.397545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.397547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.397705 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.397959 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.398030 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.398133 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:20:30.398497 16440 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } attempt: 3
I20260620 20:20:30.401185 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: Bootstrap starting.
I20260620 20:20:30.401438 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986830, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
I20260620 20:20:30.403844 16515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:30.404692 16515 log.cc:824] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:30.406811 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.398811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55348)
0620 20:20:30.398906 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.398919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.398980 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.399358 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.399360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.399532 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.399617 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.399625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.400726 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.400729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.401293 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.401304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.401813 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.401816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.401897 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.402311 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.402324 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.403493 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.405416 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.405419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.405421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.405598 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.405885 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.405888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.405889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.405967 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.406390 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.406393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.406473 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.406568 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.406674 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 20:20:30.407022 16515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: No bootstrap required, opened a new log
I20260620 20:20:30.407375 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.399053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.2.235.252:37585)
0620 20:20:30.399140 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.399144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.399162 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:20:30.399187 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.399189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.399245 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.399882 (+ 637us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.399970 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.400416 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.401439 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.401711 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.401811 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.403623 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.403626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.403628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.403653 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.405299 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.405692 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.405696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.405698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.405772 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.406068 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.406071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.406073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.406257 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.406344 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.406993 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.407089 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:20:30.410140 16440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: Bootstrap starting.
I20260620 20:20:30.410893 16515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.411137 16515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:30.411191 16515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a17155ac90b49e3b5ee448918231248, State: Initialized, Role: FOLLOWER
I20260620 20:20:30.411677 16515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.412417 16529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.412563 16529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.413038 16515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:30.415417 16440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:30.415843 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.404068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.2.235.252:37585)
0620 20:20:30.404725 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.404733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.404975 (+ 242us) server_negotiation.cc:407] Connection header received
0620 20:20:30.405113 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.405116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.405152 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.405227 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.406687 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.407564 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.408574 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.408929 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.409022 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.411555 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.411558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.411561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.411583 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.413305 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.413791 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.413794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.413796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.413956 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.414463 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.414466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.414468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.414670 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.414847 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.415558 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.415698 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":92,"threads_started":1}
I20260620 20:20:30.416976 16440 log.cc:824] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:30.417485 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.403973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.2.235.253:44627)
0620 20:20:30.404570 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.404576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.406195 (+ 1619us) server_negotiation.cc:407] Connection header received
0620 20:20:30.408682 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.408685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.408721 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.408763 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.409643 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.410313 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.411354 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.411650 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.411718 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.413437 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.413441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.413443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.413465 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.415368 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.415869 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.415873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.415875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.415962 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.416282 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.416286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.416287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.416481 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.416617 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.417215 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.417348 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
I20260620 20:20:30.419068 16440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: No bootstrap required, opened a new log
I20260620 20:20:30.416889 16446 master_runner.cc:186] Error getting master registration for 127.2.235.254:43505: OK, instance_id { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" instance_seqno: 1781986829966380 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:20:30.419796 16446 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:20:30.422231 16540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:20:30.422298 16540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:20:30.422914 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.403537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.2.235.254:43505)
0620 20:20:30.403633 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.403638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.405541 (+ 1903us) server_negotiation.cc:407] Connection header received
0620 20:20:30.405643 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.405645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.405685 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.405756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.406592 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.407331 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.410579 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.410894 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.410984 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.419243 (+ 8259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.419246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.419249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.419272 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.421020 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.421533 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.421536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.421538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.421619 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.421918 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.421922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.421924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.422131 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.422229 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.422672 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.422778 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:20:30.422901 16365 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } attempt: 3
I20260620 20:20:30.423663 16440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.423836 16440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:30.423883 16440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a15bb59ee042208ec64fdf45a7fb05, State: Initialized, Role: FOLLOWER
I20260620 20:20:30.424356 16440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.424800 16543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.424970 16543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.425596 16440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 20:20:30 dist-test-slave-8z53 krb5kdc[16284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986829, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
W20260620 20:20:30.430853 16554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:20:30.430949 16554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:20:30.432453 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.423235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:37585 (local address 127.0.0.1:55364)
0620 20:20:30.423355 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:30.423370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:30.423437 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:30.424138 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:30.424141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:30.424296 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:30.424393 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.424400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.425487 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.425491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.426306 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:30.426317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.427014 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.427018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:30.427104 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.427514 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:30.427527 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:30.428837 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:30.430949 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.430953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.430955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.431180 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.431479 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:30.431482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:30.431484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:30.431571 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:30.432002 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:30.432005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:30.432093 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:20:30.432191 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:20:30.432304 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:20:30.432863 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:30.423816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.2.235.252:37585)
0620 20:20:30.423925 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:30.423929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:30.423941 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:20:30.423973 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:30.423977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:30.424032 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:30.424123 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:30.424536 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.425316 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.426462 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:30.426806 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:30.426901 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:30.428972 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:30.428975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:30.428978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:30.428999 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:30.430787 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.431290 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.431294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.431295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.431374 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:30.431670 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:30.431673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:30.431675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:30.431858 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:30.431958 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:30.432619 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 20:20:30.432722 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 20:20:30.429201 16372 master_runner.cc:186] Error getting master registration for 127.2.235.254:43505: OK, instance_id { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" instance_seqno: 1781986829966380 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:20:30.435424 16372 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:20:30.436975 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: Bootstrap starting.
I20260620 20:20:30.439307 16365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:30.440126 16365 log.cc:824] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:30.441843 16365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: No bootstrap required, opened a new log
I20260620 20:20:30.444674 16365 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.444890 16365 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:30.444949 16365 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d045cb665246c8bb706b4242911b77, State: Initialized, Role: FOLLOWER
I20260620 20:20:30.445410 16365 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.445803 16557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.445953 16557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.446544 16365 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:20:30.451345 16568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:20:30.451422 16568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:20:30.454025 16297 master_runner.cc:263] No leader master found from master d6d045cb665246c8bb706b4242911b77
I20260620 20:20:30.454082 16297 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:20:30.463089 16318 tablet.cc:2369] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:30.606374 16468 tablet.cc:2369] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:30.622419 16557 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:30.622571 16557 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.623399 16557 leader_election.cc:288] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13a15bb59ee042208ec64fdf45a7fb05 (127.2.235.253:44627), 5a17155ac90b49e3b5ee448918231248 (127.2.235.252:37585)
I20260620 20:20:30.623862 16414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6d045cb665246c8bb706b4242911b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a15bb59ee042208ec64fdf45a7fb05" is_pre_election: true
I20260620 20:20:30.623854 16489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6d045cb665246c8bb706b4242911b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a17155ac90b49e3b5ee448918231248" is_pre_election: true
I20260620 20:20:30.624145 16489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d045cb665246c8bb706b4242911b77 in term 0.
I20260620 20:20:30.624145 16414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d045cb665246c8bb706b4242911b77 in term 0.
I20260620 20:20:30.624583 16315 leader_election.cc:302] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13a15bb59ee042208ec64fdf45a7fb05, 5a17155ac90b49e3b5ee448918231248, d6d045cb665246c8bb706b4242911b77; no voters:
I20260620 20:20:30.624864 16557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:30.624944 16557 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:30.625006 16557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:30.626219 16557 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.626555 16557 leader_election.cc:288] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [CANDIDATE]: Term 1 election: Requested vote from peers 13a15bb59ee042208ec64fdf45a7fb05 (127.2.235.253:44627), 5a17155ac90b49e3b5ee448918231248 (127.2.235.252:37585)
I20260620 20:20:30.626860 16489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6d045cb665246c8bb706b4242911b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a17155ac90b49e3b5ee448918231248"
I20260620 20:20:30.626860 16414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6d045cb665246c8bb706b4242911b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a15bb59ee042208ec64fdf45a7fb05"
I20260620 20:20:30.626995 16489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:30.626995 16414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:30.628157 16489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d045cb665246c8bb706b4242911b77 in term 1.
I20260620 20:20:30.628158 16414 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d045cb665246c8bb706b4242911b77 in term 1.
I20260620 20:20:30.628438 16315 leader_election.cc:302] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a15bb59ee042208ec64fdf45a7fb05, d6d045cb665246c8bb706b4242911b77; no voters:
I20260620 20:20:30.628618 16557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:30.628857 16557 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [term 1 LEADER]: Becoming Leader. State: Replica: d6d045cb665246c8bb706b4242911b77, State: Running, Role: LEADER
I20260620 20:20:30.629161 16557 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } }
I20260620 20:20:30.630861 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d045cb665246c8bb706b4242911b77. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.630990 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:30.631356 16572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:30.632151 16572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:30.636343 16414 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [term 1 FOLLOWER]: Refusing update from remote peer d6d045cb665246c8bb706b4242911b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:30.636368 16489 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [term 1 FOLLOWER]: Refusing update from remote peer d6d045cb665246c8bb706b4242911b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:30.636948 16570 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:30.637156 16557 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:30.640524 16529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d045cb665246c8bb706b4242911b77. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.640558 16543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d045cb665246c8bb706b4242911b77. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.640630 16529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.640654 16543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.642863 16573 mvcc.cc:205] Tried to move back new op lower bound from 7299018058281205760 to 7299018058259435520. Current Snapshot: MvccSnapshot[applied={T|T < 7299018058281205760}]
I20260620 20:20:30.643090 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.643186 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:30.643294 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.643366 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:30.643438 16529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.643540 16529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.643702 16574 mvcc.cc:205] Tried to move back new op lower bound from 7299018058281205760 to 7299018058259435520. Current Snapshot: MvccSnapshot[applied={T|T < 7299018058281205760}]
I20260620 20:20:30.644268 16543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6d045cb665246c8bb706b4242911b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d045cb665246c8bb706b4242911b77" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43505 } } peers { permanent_uuid: "13a15bb59ee042208ec64fdf45a7fb05" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44627 } } peers { permanent_uuid: "5a17155ac90b49e3b5ee448918231248" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 37585 } } }
I20260620 20:20:30.644371 16543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:20:30.644549 16572 catalog_manager.cc:1347] Generated new cluster ID: a591d8a05eea40f3905113c1845f6e1d
I20260620 20:20:30.644604 16572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:30.662613 16572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:30.663523 16572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:30.670495 16572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d045cb665246c8bb706b4242911b77: Generated new TSK 0
I20260620 20:20:30.671180 16572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:20:31.423677 16540 catalog_manager.cc:1259] Loaded cluster ID: a591d8a05eea40f3905113c1845f6e1d
I20260620 20:20:31.423758 16540 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: loading cluster ID for follower catalog manager: success
I20260620 20:20:31.425611 16540 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: acquiring CA information for follower catalog manager: success
I20260620 20:20:31.426895 16540 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a17155ac90b49e3b5ee448918231248: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:20:31.431895 16554 catalog_manager.cc:1259] Loaded cluster ID: a591d8a05eea40f3905113c1845f6e1d
I20260620 20:20:31.431984 16554 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: loading cluster ID for follower catalog manager: success
I20260620 20:20:31.433450 16554 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: acquiring CA information for follower catalog manager: success
I20260620 20:20:31.434404 16554 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13a15bb59ee042208ec64fdf45a7fb05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:20:31.722139 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16297
I20260620 20:20:31.728268 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16372
I20260620 20:20:31.733853 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16446
2026-06-20T20:20:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3035 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:31 dist-test-slave-8z53 krb5kdc[16588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:31 dist-test-slave-8z53 krb5kdc[16588](info): set up 2 sockets
Jun 20 20:20:31 dist-test-slave-8z53 krb5kdc[16588](info): commencing operation
krb5kdc: starting...
W20260620 20:20:32.725188 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:20:32 dist-test-slave-8z53 krb5kdc[16588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:20:32.829224 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43479
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:32.935123 16601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:32.935329 16601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:32.935365 16601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:32.939308 16601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:32.939354 16601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:32.939373 16601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:32.939409 16601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:32.939440 16601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:32.944379 16601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:32.945295 16601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:32.946245 16601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:32.953871 16607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:32.953935 16601 server_base.cc:1034] running on GCE node
W20260620 20:20:32.954102 16609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:32.953941 16606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:32.954752 16601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:32.955768 16601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:32.956959 16601 hybrid_clock.cc:648] HybridClock initialized: now 1781986832956930 us; error 39 us; skew 500 ppm
Jun 20 20:20:32 dist-test-slave-8z53 krb5kdc[16588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986832, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:32.959357 16601 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:32.960112 16601 webserver.cc:460] Webserver started at http://127.2.235.254:35915/ using document root <none> and password file <none>
I20260620 20:20:32.960592 16601 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:32.960675 16601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:32.960891 16601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:32.962677 16601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "2b5793116780447d9c0050230e2eba9d"
format_stamp: "Formatted at 2026-06-20 20:20:32 on dist-test-slave-8z53"
server_key: "cf8b1254d960a26815dfc952981c0db8"
server_key_iv: "59cffdac094a80ae5d030221704ff3a8"
server_key_version: "encryptionkey@0"
I20260620 20:20:32.963198 16601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "2b5793116780447d9c0050230e2eba9d"
format_stamp: "Formatted at 2026-06-20 20:20:32 on dist-test-slave-8z53"
server_key: "cf8b1254d960a26815dfc952981c0db8"
server_key_iv: "59cffdac094a80ae5d030221704ff3a8"
server_key_version: "encryptionkey@0"
I20260620 20:20:32.966355 16601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:20:32.968472 16616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:32.969556 16601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:20:32.969681 16601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "2b5793116780447d9c0050230e2eba9d"
format_stamp: "Formatted at 2026-06-20 20:20:32 on dist-test-slave-8z53"
server_key: "cf8b1254d960a26815dfc952981c0db8"
server_key_iv: "59cffdac094a80ae5d030221704ff3a8"
server_key_version: "encryptionkey@0"
I20260620 20:20:32.969781 16601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:32.994654 16601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:32.997151 16601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:32.997356 16601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:33.003542 16601 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:43479
I20260620 20:20:33.003567 16668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:43479 every 8 connection(s)
I20260620 20:20:33.004316 16601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:33.006727 16669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:33.011710 16669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d: Bootstrap starting.
I20260620 20:20:33.013991 16669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:33.014345 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16601
I20260620 20:20:33.014453 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:20:33.014707 2991 external_mini_cluster.cc:1428] Setting key e5a1387ef34a88423ff5e378b2362792
I20260620 20:20:33.014767 16669 log.cc:824] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:33.016723 16669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d: No bootstrap required, opened a new log
I20260620 20:20:33.019881 16669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5793116780447d9c0050230e2eba9d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43479 } }
I20260620 20:20:33.020095 16669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:33.020179 16669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5793116780447d9c0050230e2eba9d, State: Initialized, Role: FOLLOWER
I20260620 20:20:33.020579 16669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5793116780447d9c0050230e2eba9d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43479 } }
I20260620 20:20:33.020731 16669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:33.020823 16669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:33.020936 16669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:33.022047 16669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5793116780447d9c0050230e2eba9d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43479 } }
I20260620 20:20:33.022392 16669 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5793116780447d9c0050230e2eba9d; no voters:
I20260620 20:20:33.022714 16669 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 20 20:20:33 dist-test-slave-8z53 krb5kdc[16588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:33.022807 16674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:33.023010 16674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [term 1 LEADER]: Becoming Leader. State: Replica: 2b5793116780447d9c0050230e2eba9d, State: Running, Role: LEADER
I20260620 20:20:33.023301 16674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5793116780447d9c0050230e2eba9d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43479 } }
I20260620 20:20:33.023805 16669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:33.024451 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5793116780447d9c0050230e2eba9d. Latest consensus state: current_term: 1 leader_uuid: "2b5793116780447d9c0050230e2eba9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5793116780447d9c0050230e2eba9d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43479 } } }
I20260620 20:20:33.024626 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:33.024912 16675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5793116780447d9c0050230e2eba9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5793116780447d9c0050230e2eba9d" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43479 } } }
I20260620 20:20:33.025005 16683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:33.025089 16675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:33.028173 16683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:33.028906 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:33.015713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.2.235.254:43479)
0620 20:20:33.016083 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:33.016097 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:33.016128 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:20:33.016716 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:33.016730 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:33.017069 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:33.017418 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:33.018178 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:33.019053 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:33.019879 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:33.020481 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:33.020648 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:33.023396 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:33.023403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:33.023418 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:33.023446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:33.025373 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:33.025832 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:33.025848 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:33.025853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:33.025972 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:33.026252 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:33.026256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:33.026258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:33.026735 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:33.026926 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:33.027704 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 20:20:33.028016 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20260620 20:20:33.031678 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:20:33.034345 16683 catalog_manager.cc:1347] Generated new cluster ID: e50dee328bc84625af5181e8c78ed888
I20260620 20:20:33.034402 16683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:33.052742 16683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:33.053618 16683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:33.061285 16683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5793116780447d9c0050230e2eba9d: Generated new TSK 0
I20260620 20:20:33.061816 16683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:20:33.086642 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16601
2026-06-20T20:20:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1351 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:33 dist-test-slave-8z53 krb5kdc[16696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:33 dist-test-slave-8z53 krb5kdc[16696](info): set up 2 sockets
Jun 20 20:20:33 dist-test-slave-8z53 krb5kdc[16696](info): commencing operation
krb5kdc: starting...
W20260620 20:20:34.073184 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:20:34.177914 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43327
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:34.275900 16709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:34.276139 16709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:34.276201 16709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:34.279908 16709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:34.279978 16709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:34.280012 16709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:34.280064 16709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:34.280099 16709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:34.285511 16709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:34.286527 16709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:34.287387 16709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:34.297833 16717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:34.297859 16715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:34.298012 16714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:34.299328 16709 server_base.cc:1034] running on GCE node
I20260620 20:20:34.299911 16709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:34.301149 16709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:34.302350 16709 hybrid_clock.cc:648] HybridClock initialized: now 1781986834302291 us; error 72 us; skew 500 ppm
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:34.305617 16709 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:34.306551 16709 webserver.cc:460] Webserver started at http://127.2.235.254:39821/ using document root <none> and password file <none>
I20260620 20:20:34.307209 16709 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:34.307312 16709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:34.307595 16709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:34.310214 16709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "8881d861b0ed4472b719208f0ee7d7a2"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "5761d06d3eb481a4d042cf4955c01f24"
server_key_iv: "454f71e2d1a93a0f8485b403928116f0"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.310937 16709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "8881d861b0ed4472b719208f0ee7d7a2"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "5761d06d3eb481a4d042cf4955c01f24"
server_key_iv: "454f71e2d1a93a0f8485b403928116f0"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.315526 16709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 20:20:34.318657 16724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.320112 16709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:20:34.320236 16709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "8881d861b0ed4472b719208f0ee7d7a2"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "5761d06d3eb481a4d042cf4955c01f24"
server_key_iv: "454f71e2d1a93a0f8485b403928116f0"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.320366 16709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:34.336654 16709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:34.340520 16709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:34.340662 16709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:34.346824 16709 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:43327
I20260620 20:20:34.347123 16776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:43327 every 8 connection(s)
I20260620 20:20:34.347715 16709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:34.350296 16777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:34.352540 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16709
I20260620 20:20:34.352684 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:20:34.352978 2991 external_mini_cluster.cc:1428] Setting key 7d4bfa47149eab8efa68e5637fea350e
I20260620 20:20:34.356166 16777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap starting.
I20260620 20:20:34.358681 16777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:34.359436 16777 log.cc:824] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:34.361522 16777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: No bootstrap required, opened a new log
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:34.365418 16777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:34.365638 16777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:34.365717 16777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8881d861b0ed4472b719208f0ee7d7a2, State: Initialized, Role: FOLLOWER
I20260620 20:20:34.366211 16777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:34.366401 16777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:34.366501 16777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:34.366592 16777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:34.367408 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.354218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.2.235.254:43327)
0620 20:20:34.354579 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:34.354593 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:34.354637 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:20:34.355223 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:34.355237 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:34.355526 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:34.355783 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:34.356651 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.357679 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.358895 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.359335 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.359464 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:34.362146 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:34.362152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:34.362166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:34.362194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:34.364472 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:34.364856 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:34.364862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:34.364869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:34.364993 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:34.365243 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:34.365263 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:34.365265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:34.365720 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:34.365913 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:34.366199 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 20:20:34.366496 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260620 20:20:34.367542 16777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:34.367893 16777 leader_election.cc:302] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8881d861b0ed4472b719208f0ee7d7a2; no voters:
I20260620 20:20:34.368188 16777 leader_election.cc:288] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:34.368281 16782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:34.368494 16782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 LEADER]: Becoming Leader. State: Replica: 8881d861b0ed4472b719208f0ee7d7a2, State: Running, Role: LEADER
I20260620 20:20:34.368781 16782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:34.369333 16777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:34.370399 16783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8881d861b0ed4472b719208f0ee7d7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } } }
I20260620 20:20:34.370523 16783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:34.370815 16784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8881d861b0ed4472b719208f0ee7d7a2. Latest consensus state: current_term: 1 leader_uuid: "8881d861b0ed4472b719208f0ee7d7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } } }
I20260620 20:20:34.370959 16784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:34.370873 16791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:34.373594 16791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:34.378556 16791 catalog_manager.cc:1347] Generated new cluster ID: 8464e37fb7324afda6a8aaa0bdebc001
I20260620 20:20:34.378609 16791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:34.395751 16791 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:34.396886 16791 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:34.403334 16791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Generated new TSK 0
I20260620 20:20:34.403957 16791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:20:34.433393 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:43327
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:34.532807 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:34.533002 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:34.533061 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:34.536928 16803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:34.536975 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:34.537070 16803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:20:34.541940 16803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:34.542728 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:34.543639 16803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:34.552479 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:34.552479 16811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:34.552486 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:34.552668 16803 server_base.cc:1034] running on GCE node
I20260620 20:20:34.553232 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:34.553817 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:34.554970 16803 hybrid_clock.cc:648] HybridClock initialized: now 1781986834554933 us; error 48 us; skew 500 ppm
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:34.557686 16803 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:20:34.558485 16803 webserver.cc:460] Webserver started at http://127.2.235.193:41949/ using document root <none> and password file <none>
I20260620 20:20:34.559032 16803 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:34.559123 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:34.559329 16803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:34.561285 16803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "4eb55acc50b8447884d17ff972549caa"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "605ff08ddaa891fa1d72b49679fd6cd2"
server_key_iv: "38e1ae9b13ba82c79a1ef7c8317bf48a"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.561839 16803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "4eb55acc50b8447884d17ff972549caa"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "605ff08ddaa891fa1d72b49679fd6cd2"
server_key_iv: "38e1ae9b13ba82c79a1ef7c8317bf48a"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.565294 16803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:34.567480 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.568645 16803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:34.568773 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "4eb55acc50b8447884d17ff972549caa"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "605ff08ddaa891fa1d72b49679fd6cd2"
server_key_iv: "38e1ae9b13ba82c79a1ef7c8317bf48a"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.568881 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:34.595316 16803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:34.598675 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:34.598913 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:34.599500 16803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:34.600348 16803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:34.600400 16803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.600472 16803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:34.600509 16803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.610399 16803 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:35791
I20260620 20:20:34.610430 16931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:35791 every 8 connection(s)
I20260620 20:20:34.611301 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:34.618749 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16803
I20260620 20:20:34.618889 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:34.619184 2991 external_mini_cluster.cc:1428] Setting key 4a75daa7f082bbd037589ebc53d746f8
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:34.625448 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.613148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:54583 (local address 127.2.235.254:43327)
0620 20:20:34.613297 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:34.613302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:34.614063 (+ 761us) server_negotiation.cc:407] Connection header received
0620 20:20:34.615251 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:34.615254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:34.615307 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:34.615405 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:34.616508 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.617073 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.618211 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.618448 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.618594 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:34.621419 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:34.621422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:34.621424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:34.621444 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:34.623117 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:34.623761 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:34.623764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:34.623765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:34.623854 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:34.624225 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:34.624228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:34.624229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:34.624388 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:34.624459 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:34.625178 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 20:20:34.625331 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:20:34.626624 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.613542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:54583)
0620 20:20:34.613940 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:34.613977 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:34.615057 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:34.615578 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:34.615585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:34.615908 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:34.616346 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:34.616368 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.617188 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.617196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:34.618095 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:34.618104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.618630 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.618634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:34.618809 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:34.619418 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:34.619433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:34.621242 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:34.623241 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:34.623246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:34.623256 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:34.623632 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:34.623946 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:34.623952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:34.623957 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:34.624131 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:34.624494 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:34.624502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:34.624906 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 20:20:34.625115 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 20:20:34.625466 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260620 20:20:34.627887 16932 heartbeater.cc:347] Connected to a master server at 127.2.235.254:43327
I20260620 20:20:34.628093 16932 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:34.628506 16932 heartbeater.cc:511] Master 127.2.235.254:43327 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:20:34.630014 16737 ts_manager.cc:194] Registered new tserver with Master: 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791)
I20260620 20:20:34.631083 16737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:54583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:20:34.644124 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:43327
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:34.743009 16937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:34.743187 16937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:34.743220 16937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:34.746656 16937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:34.746702 16937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:34.746755 16937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:20:34.751292 16937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:34.752022 16937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:34.752823 16937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:34.760921 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:34.761034 16937 server_base.cc:1034] running on GCE node
W20260620 20:20:34.760914 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:34.760913 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:34.761477 16937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:34.762003 16937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:34.763154 16937 hybrid_clock.cc:648] HybridClock initialized: now 1781986834763127 us; error 36 us; skew 500 ppm
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:34.765671 16937 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:20:34.766374 16937 webserver.cc:460] Webserver started at http://127.2.235.194:45637/ using document root <none> and password file <none>
I20260620 20:20:34.766885 16937 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:34.766968 16937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:34.767164 16937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:34.768918 16937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "75227bd7b5bf4129a64d8a30606b789e"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "4e38a656f3eedfbed589c5074b7ca7ff"
server_key_iv: "c817c4a60b9377bd92a3588e61229113"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.769495 16937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "75227bd7b5bf4129a64d8a30606b789e"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "4e38a656f3eedfbed589c5074b7ca7ff"
server_key_iv: "c817c4a60b9377bd92a3588e61229113"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.772756 16937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:20:34.774878 16952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.775933 16937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:20:34.776048 16937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "75227bd7b5bf4129a64d8a30606b789e"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "4e38a656f3eedfbed589c5074b7ca7ff"
server_key_iv: "c817c4a60b9377bd92a3588e61229113"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.776142 16937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:34.801661 16937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:34.804111 16937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:34.804275 16937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:34.804744 16937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:34.805486 16937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:34.805545 16937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.805603 16937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:34.805663 16937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.813796 16937 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:37605
I20260620 20:20:34.813891 17065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:37605 every 8 connection(s)
I20260620 20:20:34.814620 16937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:34.818944 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 16937
I20260620 20:20:34.819057 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:34.819345 2991 external_mini_cluster.cc:1428] Setting key 64128c7cd9c4f594ffa3ef2d61568dd5
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:34.828683 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.816220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:46087 (local address 127.2.235.254:43327)
0620 20:20:34.816343 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:34.816347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:34.817054 (+ 707us) server_negotiation.cc:407] Connection header received
0620 20:20:34.818481 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:34.818484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:34.818540 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:34.818636 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:34.819995 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.820612 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.821703 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.822097 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.822218 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:34.824817 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:34.824823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:34.824828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:34.824861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:34.826688 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:34.827176 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:34.827182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:34.827185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:34.827281 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:34.827583 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:34.827587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:34.827589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:34.827833 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:34.827939 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:34.828372 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 20:20:34.828496 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:20:34.829298 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:34.816491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:46087)
0620 20:20:34.816917 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:34.816957 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:34.818281 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:34.818809 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:34.818816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:34.819292 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:34.819859 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:34.819879 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.820737 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.820740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:34.821595 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:34.821604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:34.822285 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:34.822291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:34.822515 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:34.823079 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:34.823098 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:34.824643 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:34.826801 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:34.826805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:34.826815 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:34.827075 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:34.827375 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:34.827377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:34.827378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:34.827493 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:34.827929 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:34.827934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:34.828123 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 20:20:34.828263 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 20:20:34.828489 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":104,"threads_started":1}
I20260620 20:20:34.830392 17066 heartbeater.cc:347] Connected to a master server at 127.2.235.254:43327
I20260620 20:20:34.830614 17066 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:34.831223 17066 heartbeater.cc:511] Master 127.2.235.254:43327 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:20:34.832295 16737 ts_manager.cc:194] Registered new tserver with Master: 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605)
I20260620 20:20:34.832814 16737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:46087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:20:34.845445 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:43327
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:34.948575 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:34.948769 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:34.948805 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:34.952343 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:34.952394 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:34.952443 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:20:34.956887 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:34.957881 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:34.958725 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:34.966693 17077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:34.966748 17076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:34.966810 17071 server_base.cc:1034] running on GCE node
W20260620 20:20:34.966990 17079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:34.967346 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:34.967865 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:34.969033 17071 hybrid_clock.cc:648] HybridClock initialized: now 1781986834969026 us; error 40 us; skew 500 ppm
Jun 20 20:20:34 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:34.974982 17071 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:20:34.975656 17071 webserver.cc:460] Webserver started at http://127.2.235.195:36661/ using document root <none> and password file <none>
I20260620 20:20:34.976125 17071 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:34.976174 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:34.976305 17071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:34.978165 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "b0452d636287495281bfc12df81f04ed"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "b10b00b4528d18bc8f333f6bb8b534e4"
server_key_iv: "d762b9206cbff59c8693881d2bf32f15"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.978650 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "b0452d636287495281bfc12df81f04ed"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "b10b00b4528d18bc8f333f6bb8b534e4"
server_key_iv: "d762b9206cbff59c8693881d2bf32f15"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.981840 17071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:20:34.983969 17086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:34.985067 17071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:20:34.985146 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "b0452d636287495281bfc12df81f04ed"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "b10b00b4528d18bc8f333f6bb8b534e4"
server_key_iv: "d762b9206cbff59c8693881d2bf32f15"
server_key_version: "encryptionkey@0"
I20260620 20:20:34.985285 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:35.008955 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:35.011368 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:35.011538 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:35.012068 17071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:35.012774 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:35.012817 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:35.012894 17071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:35.012931 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:35.021134 17071 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:34429
I20260620 20:20:35.021154 17199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:34429 every 8 connection(s)
I20260620 20:20:35.021888 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986834, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:35.030222 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17071
I20260620 20:20:35.030337 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:35.030565 2991 external_mini_cluster.cc:1428] Setting key 9b212a9e78a73296a5191541929f1ece
I20260620 20:20:35.033535 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.023320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:43441 (local address 127.2.235.254:43327)
0620 20:20:35.023441 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.023445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.024101 (+ 656us) server_negotiation.cc:407] Connection header received
0620 20:20:35.025217 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.025220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.025298 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.025395 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:35.026406 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.026952 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.028021 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.028222 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.028315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.030449 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:35.030452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:35.030454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:35.030473 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:35.031958 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.032392 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.032395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.032397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.032479 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.032740 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.032742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.032743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.032883 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:35.032949 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.033344 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.033434 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:20:35.034189 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.023626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:43441)
0620 20:20:35.023985 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.024020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.025032 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.025542 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.025548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.025883 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:35.026272 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.026283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.027070 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.027073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.027912 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.027920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.028353 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.028357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.028497 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.028967 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:35.028981 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:35.030238 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:35.032051 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.032055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.032063 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.032304 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.032555 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.032557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.032559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.032673 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.032956 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:35.032960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:35.033134 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.033310 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.033517 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260620 20:20:35.035141 17200 heartbeater.cc:347] Connected to a master server at 127.2.235.254:43327
I20260620 20:20:35.035336 17200 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:35.035755 17200 heartbeater.cc:511] Master 127.2.235.254:43327 requested a full tablet report, sending...
I20260620 20:20:35.036631 16737 ts_manager.cc:194] Registered new tserver with Master: b0452d636287495281bfc12df81f04ed (127.2.235.195:34429)
I20260620 20:20:35.037143 16737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:43441
I20260620 20:20:35.043749 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:35.052357 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.045565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.2.235.254:43327)
0620 20:20:35.045685 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.045691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.045884 (+ 193us) server_negotiation.cc:407] Connection header received
0620 20:20:35.046023 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.046025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.046066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.046136 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:35.046785 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.047253 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.048056 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.048253 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.048351 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.049577 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:35.049580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:35.049583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:35.049604 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:35.050838 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.051204 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.051208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.051210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.051276 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.051529 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.051533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.051535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.051718 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:35.051845 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.052073 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.052207 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:20:35.054788 16737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:35.057564 16737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:35.070441 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.065199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:34429 (local address 127.0.0.1:49808)
0620 20:20:35.065758 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.065769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.065882 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.066340 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.066346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.066379 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.066662 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.066668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.067732 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.067738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.068928 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.068939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.069937 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.069943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.070038 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.070075 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.070178 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.070276 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":97,"spinlock_wait_cycles":5248,"thread_start_us":106,"threads_started":1}
I20260620 20:20:35.070629 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.065349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.2.235.195:34429)
0620 20:20:35.065707 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.065717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.065930 (+ 213us) server_negotiation.cc:407] Connection header received
0620 20:20:35.065999 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.066003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.066185 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.066325 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.066769 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.067600 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.069043 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.069826 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.069921 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.070038 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.070212 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.070299 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260620 20:20:35.070883 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.064606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:35791 (local address 127.0.0.1:43448)
0620 20:20:35.065218 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.065894 (+ 676us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.065961 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.066537 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.066540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.066558 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.066736 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.066743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.068102 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.068105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.069218 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.069228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.070494 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.070497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.070613 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.070634 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.070700 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.070764 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":71,"thread_start_us":65,"threads_started":1}
I20260620 20:20:35.071213 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.064761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.2.235.193:35791)
0620 20:20:35.066133 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.066143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.066161 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:20:35.066234 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.066241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.066400 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.066554 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.066854 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.067981 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.069401 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.070204 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.070294 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.070451 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.071029 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.071123 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":111,"threads_started":1}
I20260620 20:20:35.071612 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.064782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:37605 (local address 127.0.0.1:33050)
0620 20:20:35.065505 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.065561 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.065657 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.066343 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.066352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.066381 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.066647 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.066656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.068085 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.068088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.069251 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.069288 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.070473 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.070479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.070569 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.071353 (+ 784us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.071422 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.071511 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":936,"spinlock_wait_cycles":4608,"thread_start_us":91,"threads_started":1}
I20260620 20:20:35.071651 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.065170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.2.235.194:37605)
0620 20:20:35.065610 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.065621 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.065644 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:20:35.065835 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.065840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.066027 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.066179 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.066781 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.067955 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.069406 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.070091 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.070198 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.070314 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.071467 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.071553 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":109,"threads_started":1}
I20260620 20:20:35.073108 17134 tablet_service.cc:1467] Processing CreateTablet for tablet 3398ab79019046339ee63baa934ad072 (DEFAULT_TABLE table=test-table [id=6f92f936b50047cb8707db3e69f76c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:35.074122 17134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3398ab79019046339ee63baa934ad072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:35.073899 17000 tablet_service.cc:1467] Processing CreateTablet for tablet 3398ab79019046339ee63baa934ad072 (DEFAULT_TABLE table=test-table [id=6f92f936b50047cb8707db3e69f76c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:35.074873 17000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3398ab79019046339ee63baa934ad072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:35.075753 16866 tablet_service.cc:1467] Processing CreateTablet for tablet 3398ab79019046339ee63baa934ad072 (DEFAULT_TABLE table=test-table [id=6f92f936b50047cb8707db3e69f76c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:35.076663 16866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3398ab79019046339ee63baa934ad072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:35.079361 17217 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Bootstrap starting.
I20260620 20:20:35.081171 17217 tablet_bootstrap.cc:654] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:35.082082 17218 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Bootstrap starting.
I20260620 20:20:35.082970 17217 log.cc:824] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:35.083062 17219 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Bootstrap starting.
I20260620 20:20:35.084263 17218 tablet_bootstrap.cc:654] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:35.084941 17219 tablet_bootstrap.cc:654] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:35.085013 17218 log.cc:824] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:35.085100 17217 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: No bootstrap required, opened a new log
I20260620 20:20:35.085336 17217 ts_tablet_manager.cc:1397] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:20:35.086103 17219 log.cc:824] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:35.086776 17218 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: No bootstrap required, opened a new log
I20260620 20:20:35.087003 17218 ts_tablet_manager.cc:1397] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:20:35.088055 17219 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: No bootstrap required, opened a new log
I20260620 20:20:35.088263 17219 ts_tablet_manager.cc:1397] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 20:20:35.088795 17217 raft_consensus.cc:348] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.089073 17217 raft_consensus.cc:374] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:35.089155 17217 raft_consensus.cc:729] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0452d636287495281bfc12df81f04ed, State: Initialized, Role: FOLLOWER
I20260620 20:20:35.089546 17217 consensus_queue.cc:260] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.090078 17200 heartbeater.cc:503] Master 127.2.235.254:43327 was elected leader, sending a full tablet report...
I20260620 20:20:35.090449 17217 ts_tablet_manager.cc:1428] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:20:35.090756 17218 raft_consensus.cc:348] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.091012 17218 raft_consensus.cc:374] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:35.091120 17218 raft_consensus.cc:729] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75227bd7b5bf4129a64d8a30606b789e, State: Initialized, Role: FOLLOWER
I20260620 20:20:35.091503 17218 consensus_queue.cc:260] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.091800 17219 raft_consensus.cc:348] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.092175 17219 raft_consensus.cc:374] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:35.092262 17219 raft_consensus.cc:729] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb55acc50b8447884d17ff972549caa, State: Initialized, Role: FOLLOWER
I20260620 20:20:35.092334 17218 ts_tablet_manager.cc:1428] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:20:35.092707 17066 heartbeater.cc:503] Master 127.2.235.254:43327 was elected leader, sending a full tablet report...
I20260620 20:20:35.092916 17219 consensus_queue.cc:260] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.093559 16932 heartbeater.cc:503] Master 127.2.235.254:43327 was elected leader, sending a full tablet report...
I20260620 20:20:35.093845 17219 ts_tablet_manager.cc:1428] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
W20260620 20:20:35.112349 16933 tablet.cc:2369] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:35.272719 17201 tablet.cc:2369] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:35.315703 17067 tablet.cc:2369] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:35.330159 17224 raft_consensus.cc:482] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:35.330313 17224 raft_consensus.cc:504] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.330626 17225 raft_consensus.cc:482] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:35.330789 17225 raft_consensus.cc:504] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.331617 17224 leader_election.cc:288] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791), b0452d636287495281bfc12df81f04ed (127.2.235.195:34429)
I20260620 20:20:35.333213 17225 leader_election.cc:288] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605), b0452d636287495281bfc12df81f04ed (127.2.235.195:34429)
I20260620 20:20:35.336983 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.331849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:34429 (local address 127.2.235.194:53287)
0620 20:20:35.332326 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.332340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.332435 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.332842 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.332845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.332881 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.333025 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.333031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.334142 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.334145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.335611 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.335623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.336514 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.336657 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.336679 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.336752 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.336814 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":144,"spinlock_wait_cycles":4096,"thread_start_us":71,"threads_started":1}
I20260620 20:20:35.337005 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.332322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:53287 (local address 127.2.235.195:34429)
0620 20:20:35.332467 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.332472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.332486 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:20:35.332551 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.332556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.332607 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.332711 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.333161 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.333991 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.335762 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.336363 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336491 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.336541 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.336786 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.336874 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":65}
I20260620 20:20:35.337005 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.332074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:55659 (local address 127.2.235.194:37605)
0620 20:20:35.332187 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.332193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.332732 (+ 539us) server_negotiation.cc:407] Connection header received
0620 20:20:35.332935 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.332938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.332983 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.333035 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.333469 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.334209 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.335641 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.336197 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336277 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.336327 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.336837 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.336904 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:20:35.337678 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.332168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:58757 (local address 127.2.235.193:35791)
0620 20:20:35.332266 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.332270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.332285 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:20:35.332472 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.332476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.332536 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.332588 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.333150 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.333978 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.335762 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.336358 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.336542 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.337480 (+ 938us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.337555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":29}
I20260620 20:20:35.337718 17154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "75227bd7b5bf4129a64d8a30606b789e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452d636287495281bfc12df81f04ed" is_pre_election: true
I20260620 20:20:35.337693 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.332638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:34429 (local address 127.2.235.193:57933)
0620 20:20:35.333484 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.333500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.333571 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.334019 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.334022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.334036 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.334129 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.334134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.335158 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.335161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.336291 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.336299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.337343 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.337356 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.337471 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.337495 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.337537 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.337584 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":108,"threads_started":1}
I20260620 20:20:35.338013 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.331981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:37605 (local address 127.2.235.193:55659)
0620 20:20:35.332639 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.332653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.332803 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.333174 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.333177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.333196 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.333339 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.333346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.334323 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.334325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.335499 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.335508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.336418 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.336675 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.336689 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.337601 (+ 912us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.337683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":152,"thread_start_us":92,"threads_started":1}
I20260620 20:20:35.338029 17154 raft_consensus.cc:2454] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75227bd7b5bf4129a64d8a30606b789e in term 0.
I20260620 20:20:35.338075 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.333018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:57933 (local address 127.2.235.195:34429)
0620 20:20:35.333801 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.333809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.333821 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:20:35.333860 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.333863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.333900 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.334215 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:35.334310 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.335042 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336434 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.337188 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.337785 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.337844 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.337879 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.337944 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":719,"thread_start_us":110,"threads_started":1}
I20260620 20:20:35.338255 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.331824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:35791 (local address 127.2.235.194:58757)
0620 20:20:35.332185 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.332202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.332307 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.332804 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.332809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.332829 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:35.333004 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.333013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.334174 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.334177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.335611 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.335623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.336519 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.336523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.337347 (+ 824us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.337370 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.338053 (+ 683us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.338129 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":798,"spinlock_wait_cycles":9600,"thread_start_us":84,"threads_started":1}
I20260620 20:20:35.338583 17154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "4eb55acc50b8447884d17ff972549caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452d636287495281bfc12df81f04ed" is_pre_election: true
I20260620 20:20:35.338646 16953 leader_election.cc:302] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed; no voters:
I20260620 20:20:35.338726 17154 raft_consensus.cc:2454] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb55acc50b8447884d17ff972549caa in term 0.
I20260620 20:20:35.338801 16886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "75227bd7b5bf4129a64d8a30606b789e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb55acc50b8447884d17ff972549caa" is_pre_election: true
I20260620 20:20:35.339002 16886 raft_consensus.cc:2454] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75227bd7b5bf4129a64d8a30606b789e in term 0.
I20260620 20:20:35.339013 17224 raft_consensus.cc:2781] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:35.339035 16819 leader_election.cc:302] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb55acc50b8447884d17ff972549caa, b0452d636287495281bfc12df81f04ed; no voters:
I20260620 20:20:35.339095 17224 raft_consensus.cc:482] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:35.339136 17224 raft_consensus.cc:3037] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:35.339231 17020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "4eb55acc50b8447884d17ff972549caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75227bd7b5bf4129a64d8a30606b789e" is_pre_election: true
I20260620 20:20:35.339288 17225 raft_consensus.cc:2781] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:35.339367 17225 raft_consensus.cc:482] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:35.339417 17225 raft_consensus.cc:3037] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:35.340250 17224 raft_consensus.cc:504] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.340487 17020 raft_consensus.cc:2379] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4eb55acc50b8447884d17ff972549caa in current term 1: Already voted for candidate 75227bd7b5bf4129a64d8a30606b789e in this term.
I20260620 20:20:35.340451 17225 raft_consensus.cc:504] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.340610 17224 leader_election.cc:288] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [CANDIDATE]: Term 1 election: Requested vote from peers 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791), b0452d636287495281bfc12df81f04ed (127.2.235.195:34429)
I20260620 20:20:35.340806 17225 leader_election.cc:288] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 1 election: Requested vote from peers 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605), b0452d636287495281bfc12df81f04ed (127.2.235.195:34429)
I20260620 20:20:35.340963 16886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "75227bd7b5bf4129a64d8a30606b789e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb55acc50b8447884d17ff972549caa"
I20260620 20:20:35.341058 17154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "75227bd7b5bf4129a64d8a30606b789e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452d636287495281bfc12df81f04ed"
I20260620 20:20:35.341122 16886 raft_consensus.cc:2379] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75227bd7b5bf4129a64d8a30606b789e in current term 1: Already voted for candidate 4eb55acc50b8447884d17ff972549caa in this term.
I20260620 20:20:35.341156 17154 raft_consensus.cc:3037] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:35.341403 17153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "4eb55acc50b8447884d17ff972549caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0452d636287495281bfc12df81f04ed"
I20260620 20:20:35.341878 17020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3398ab79019046339ee63baa934ad072" candidate_uuid: "4eb55acc50b8447884d17ff972549caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75227bd7b5bf4129a64d8a30606b789e"
I20260620 20:20:35.342005 17020 raft_consensus.cc:2379] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4eb55acc50b8447884d17ff972549caa in current term 1: Already voted for candidate 75227bd7b5bf4129a64d8a30606b789e in this term.
I20260620 20:20:35.342244 16819 leader_election.cc:302] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb55acc50b8447884d17ff972549caa; no voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed
I20260620 20:20:35.342378 17154 raft_consensus.cc:2454] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75227bd7b5bf4129a64d8a30606b789e in term 1.
I20260620 20:20:35.342409 17225 raft_consensus.cc:2726] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 20:20:35.342638 16953 leader_election.cc:302] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed; no voters: 4eb55acc50b8447884d17ff972549caa
I20260620 20:20:35.342795 17224 raft_consensus.cc:2781] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:35.342995 17224 raft_consensus.cc:686] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 LEADER]: Becoming Leader. State: Replica: 75227bd7b5bf4129a64d8a30606b789e, State: Running, Role: LEADER
I20260620 20:20:35.343513 17224 consensus_queue.cc:237] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.346206 16740 catalog_manager.cc:5495] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194). New cstate: current_term: 1 leader_uuid: "75227bd7b5bf4129a64d8a30606b789e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } health_report { overall_health: UNKNOWN } } }
I20260620 20:20:35.362826 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.359602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.2.235.194:37605)
0620 20:20:35.359701 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.359705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.359786 (+ 81us) server_negotiation.cc:407] Connection header received
0620 20:20:35.359943 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.359945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.359985 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.360073 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:35.360287 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.360730 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.361566 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.361747 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.361846 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.362114 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:35.362464 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:35.362601 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.362660 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.362705 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:20:35.367916 17154 raft_consensus.cc:1261] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Refusing update from remote peer 75227bd7b5bf4129a64d8a30606b789e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:35.367928 16886 raft_consensus.cc:1261] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Refusing update from remote peer 75227bd7b5bf4129a64d8a30606b789e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:35.368446 17231 consensus_queue.cc:1035] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:35.368634 17224 consensus_queue.cc:1035] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:35.374948 17236 mvcc.cc:205] Tried to move back new op lower bound from 7299018077661143040 to 7299018077569290240. Current Snapshot: MvccSnapshot[applied={T|T < 7299018077661143040}]
I20260620 20:20:35.375237 17234 mvcc.cc:205] Tried to move back new op lower bound from 7299018077661143040 to 7299018077569290240. Current Snapshot: MvccSnapshot[applied={T|T < 7299018077661143040}]
I20260620 20:20:35.378211 17235 mvcc.cc:205] Tried to move back new op lower bound from 7299018077661143040 to 7299018077569290240. Current Snapshot: MvccSnapshot[applied={T|T < 7299018077661143040}]
I20260620 20:20:35.383008 16737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:35.383116 16737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:35.385130 16737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Sending DeleteTablet for 3 replicas of tablet 3398ab79019046339ee63baa934ad072
I20260620 20:20:35.385855 16866 tablet_service.cc:1514] Processing DeleteTablet for tablet 3398ab79019046339ee63baa934ad072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:35 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:43448
I20260620 20:20:35.385958 17000 tablet_service.cc:1514] Processing DeleteTablet for tablet 3398ab79019046339ee63baa934ad072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:35 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33050
I20260620 20:20:35.385953 17134 tablet_service.cc:1514] Processing DeleteTablet for tablet 3398ab79019046339ee63baa934ad072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:35 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:49808
I20260620 20:20:35.386379 17245 tablet_replica.cc:330] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: stopping tablet replica
I20260620 20:20:35.386472 17244 tablet_replica.cc:330] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: stopping tablet replica
I20260620 20:20:35.386571 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16803
W20260620 20:20:35.392901 16726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:35791 (error 108)
I20260620 20:20:35.392972 17245 raft_consensus.cc:2229] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:35.393115 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16937
I20260620 20:20:35.394273 17246 tablet_replica.cc:330] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: stopping tablet replica
I20260620 20:20:35.394573 17246 raft_consensus.cc:2229] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:35.394634 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.394377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:35791 (local address 127.0.0.1:43458)
0620 20:20:35.394494 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.394569 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:35791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 20:20:35.394784 16726 catalog_manager.cc:4584] TS 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3398ab79019046339ee63baa934ad072: Network error: Client connection negotiation failed: client connection to 127.2.235.193:35791: connect: Connection refused (error 111)
I20260620 20:20:35.394861 17246 raft_consensus.cc:2258] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:35.396109 17246 ts_tablet_manager.cc:1905] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:35.398311 17246 ts_tablet_manager.cc:1918] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:35.398442 17246 log.cc:1196] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/3398ab79019046339ee63baa934ad072
I20260620 20:20:35.398769 17246 ts_tablet_manager.cc:1939] T 3398ab79019046339ee63baa934ad072 P b0452d636287495281bfc12df81f04ed: Deleting consensus metadata
I20260620 20:20:35.399897 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 17071
I20260620 20:20:35.400076 16725 catalog_manager.cc:4841] TS b0452d636287495281bfc12df81f04ed (127.2.235.195:34429): tablet 3398ab79019046339ee63baa934ad072 (table test-table [id=6f92f936b50047cb8707db3e69f76c1e]) successfully deleted
I20260620 20:20:35.400880 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.399937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:37605 (local address 127.0.0.1:33056)
0620 20:20:35.400780 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.400825 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.194:37605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":781}
I20260620 20:20:35.405287 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 16709
I20260620 20:20:35.411473 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43327
--webserver_interface=127.2.235.254
--webserver_port=39821
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:35.511632 17248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:35.511842 17248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:35.511875 17248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:35.515650 17248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:35.515692 17248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:35.515710 17248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:35.515748 17248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:35.515777 17248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:35.520505 17248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:35.521320 17248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:35.522146 17248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:35.533479 17257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:35.533524 17248 server_base.cc:1034] running on GCE node
W20260620 20:20:35.533460 17255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:35.533499 17254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:35.534198 17248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:35.535207 17248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:35.536373 17248 hybrid_clock.cc:648] HybridClock initialized: now 1781986835536341 us; error 29 us; skew 500 ppm
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:35.538908 17248 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:35.539670 17248 webserver.cc:460] Webserver started at http://127.2.235.254:39821/ using document root <none> and password file <none>
I20260620 20:20:35.540166 17248 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:35.540251 17248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:35.543690 17248 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:35.545712 17264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:35.546806 17248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:35.546932 17248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "8881d861b0ed4472b719208f0ee7d7a2"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "5761d06d3eb481a4d042cf4955c01f24"
server_key_iv: "454f71e2d1a93a0f8485b403928116f0"
server_key_version: "encryptionkey@0"
I20260620 20:20:35.547461 17248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:35.571760 17248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:35.574666 17248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:35.574841 17248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:35.581565 17248 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:43327
I20260620 20:20:35.581879 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:43327 every 8 connection(s)
I20260620 20:20:35.582579 17248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:35.586562 17317 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:20:35.587090 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17248
I20260620 20:20:35.587502 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:35791
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:43327
--webserver_port=41949
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:35.588323 17317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap starting.
I20260620 20:20:35.610761 17317 log.cc:824] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:35.621863 17317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:35.622295 17317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap complete.
I20260620 20:20:35.627808 17317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:35.628275 17317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8881d861b0ed4472b719208f0ee7d7a2, State: Initialized, Role: FOLLOWER
I20260620 20:20:35.628791 17317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:35.628950 17317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:35.629030 17317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:35.629144 17317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:20:35.631608 17317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:35.631950 17317 leader_election.cc:302] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8881d861b0ed4472b719208f0ee7d7a2; no voters:
I20260620 20:20:35.632274 17317 leader_election.cc:288] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:20:35.632371 17321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:20:35.632612 17321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 LEADER]: Becoming Leader. State: Replica: 8881d861b0ed4472b719208f0ee7d7a2, State: Running, Role: LEADER
I20260620 20:20:35.632920 17321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:35.633479 17317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:35.634255 17323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8881d861b0ed4472b719208f0ee7d7a2. Latest consensus state: current_term: 2 leader_uuid: "8881d861b0ed4472b719208f0ee7d7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } } }
I20260620 20:20:35.634253 17322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8881d861b0ed4472b719208f0ee7d7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } } }
I20260620 20:20:35.634389 17323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:35.634389 17322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:35.635558 17330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:35.639838 17330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:35.640425 17330 catalog_manager.cc:1259] Loaded cluster ID: 8464e37fb7324afda6a8aaa0bdebc001
I20260620 20:20:35.640482 17330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:35.641762 17330 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 20:20:35.646194 17330 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 20:20:35.651067 17330 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:35.651665 17330 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Loaded TSK: 0
I20260620 20:20:35.652783 17330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:20:35.707088 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:35.707324 17319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:35.707393 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:35.711238 17319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:35.711304 17319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:35.711390 17319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:20:35.716068 17319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:35791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=41949
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:35.716954 17319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:35.717892 17319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:35.727301 17344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:35.727406 17345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:35.727308 17347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:35.727679 17319 server_base.cc:1034] running on GCE node
I20260620 20:20:35.728087 17319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:35.728741 17319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:35.729934 17319 hybrid_clock.cc:648] HybridClock initialized: now 1781986835729912 us; error 29 us; skew 500 ppm
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:35.733073 17319 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:20:35.734133 17319 webserver.cc:460] Webserver started at http://127.2.235.193:41949/ using document root <none> and password file <none>
I20260620 20:20:35.734738 17319 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:35.734838 17319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:35.739387 17319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:20:35.741819 17354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:35.743147 17319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:20:35.743265 17319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "4eb55acc50b8447884d17ff972549caa"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "605ff08ddaa891fa1d72b49679fd6cd2"
server_key_iv: "38e1ae9b13ba82c79a1ef7c8317bf48a"
server_key_version: "encryptionkey@0"
I20260620 20:20:35.743841 17319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:35.766552 17319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:35.769811 17319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:35.769990 17319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:35.770681 17319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:35.771950 17361 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:20:35.775452 17319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:20:35.775514 17319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 20:20:35.775573 17319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:20:35.777561 17319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:20:35.777616 17319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:35.777913 17361 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Bootstrap starting.
I20260620 20:20:35.787789 17319 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:35791
I20260620 20:20:35.787889 17468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:35791 every 8 connection(s)
I20260620 20:20:35.788661 17319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:35.793959 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17319
I20260620 20:20:35.794368 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:37605
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:43327
--webserver_port=45637
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:35.795050 17361 log.cc:824] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:35.803364 17361 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:35.803815 17361 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Bootstrap complete.
I20260620 20:20:35.804633 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.790566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:39343)
0620 20:20:35.791101 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.791137 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.791870 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.792849 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.792856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.793221 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:35.793664 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.793676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.795237 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.795240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.796347 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.796358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.797080 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.797084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.797317 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.798028 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:35.798052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:35.799805 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:35.801868 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.801873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.801885 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.802121 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.802456 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.802459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.802461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.802601 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.803107 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:35.803113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:35.803334 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.803538 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.803844 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":170,"threads_started":1}
I20260620 20:20:35.804621 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.790503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:39343 (local address 127.2.235.254:43327)
0620 20:20:35.790828 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.790843 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.791235 (+ 392us) server_negotiation.cc:407] Connection header received
0620 20:20:35.792239 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.792249 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.792514 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.792795 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:35.793811 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.794903 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.796470 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.796942 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.797110 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.799988 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:35.799993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:35.800003 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:35.800028 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:35.801751 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.802228 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.802234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.802239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.802369 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.802689 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.802694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.802695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.802999 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:35.803133 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.803654 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.803898 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260620 20:20:35.805171 17361 ts_tablet_manager.cc:1397] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.004s
I20260620 20:20:35.806756 17469 heartbeater.cc:347] Connected to a master server at 127.2.235.254:43327
I20260620 20:20:35.806965 17469 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:35.807384 17469 heartbeater.cc:511] Master 127.2.235.254:43327 requested a full tablet report, sending...
I20260620 20:20:35.808748 17361 raft_consensus.cc:348] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.809374 17361 raft_consensus.cc:729] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb55acc50b8447884d17ff972549caa, State: Initialized, Role: FOLLOWER
I20260620 20:20:35.809566 17281 ts_manager.cc:194] Registered new tserver with Master: 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791)
I20260620 20:20:35.809845 17361 consensus_queue.cc:260] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:35.810801 17361 ts_tablet_manager.cc:1428] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20260620 20:20:35.812819 17281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:39343
I20260620 20:20:35.814947 17469 heartbeater.cc:503] Master 127.2.235.254:43327 was elected leader, sending a full tablet report...
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:20:35.822690 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.812418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.2.235.193:35791)
0620 20:20:35.812726 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.812736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.813226 (+ 490us) server_negotiation.cc:407] Connection header received
0620 20:20:35.813448 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.813454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.813619 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.813740 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:35.814375 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.815336 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.816572 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.816913 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.816993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.819357 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:35.819365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:35.819372 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:35.819390 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:35.820813 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.821439 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.821443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.821444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.821514 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.821805 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.821807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.821808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.822019 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:35.822110 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.822463 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.822585 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
I20260620 20:20:35.822739 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.812364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:35791 (local address 127.0.0.1:43468)
0620 20:20:35.813096 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.813136 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.813283 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.813757 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.813763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.814136 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:35.814252 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.814260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.815443 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.815450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.816449 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.816461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.817032 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.817037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.817152 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.817660 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:35.817677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:35.819206 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:35.820923 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.820949 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.820954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.821283 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.821601 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.821605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.821607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.821710 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.822119 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:35.822125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:35.822244 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.822392 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.822573 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":103,"threads_started":1}
I20260620 20:20:35.823544 17403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3398ab79019046339ee63baa934ad072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:35 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:43468
I20260620 20:20:35.824218 17478 tablet_replica.cc:330] stopping tablet replica
I20260620 20:20:35.824476 17478 raft_consensus.cc:2229] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:35.824697 17478 raft_consensus.cc:2258] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:35.825582 17478 ts_tablet_manager.cc:1905] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:35.828821 17478 ts_tablet_manager.cc:1918] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:35.828903 17478 log.cc:1196] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/3398ab79019046339ee63baa934ad072
I20260620 20:20:35.829169 17478 ts_tablet_manager.cc:1939] T 3398ab79019046339ee63baa934ad072 P 4eb55acc50b8447884d17ff972549caa: Deleting consensus metadata
I20260620 20:20:35.830621 17266 catalog_manager.cc:4841] TS 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791): tablet 3398ab79019046339ee63baa934ad072 (table test-table [id=6f92f936b50047cb8707db3e69f76c1e]) successfully deleted
W20260620 20:20:35.907388 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:35.907600 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:35.907642 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:35.911559 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:35.911608 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:35.911660 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:20:35.915931 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:37605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=45637
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:35.916630 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:35.917501 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:35.926195 17483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:35.926194 17484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:35.926326 17486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:35.926543 17474 server_base.cc:1034] running on GCE node
I20260620 20:20:35.926931 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:35.927474 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:35.928624 17474 hybrid_clock.cc:648] HybridClock initialized: now 1781986835928617 us; error 27 us; skew 500 ppm
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:35.931170 17474 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:20:35.931903 17474 webserver.cc:460] Webserver started at http://127.2.235.194:45637/ using document root <none> and password file <none>
I20260620 20:20:35.932399 17474 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:35.932479 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:35.935988 17474 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:20:35.937952 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:35.939002 17474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:20:35.939118 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "75227bd7b5bf4129a64d8a30606b789e"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "4e38a656f3eedfbed589c5074b7ca7ff"
server_key_iv: "c817c4a60b9377bd92a3588e61229113"
server_key_version: "encryptionkey@0"
I20260620 20:20:35.939559 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:35.960762 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:35.963593 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:35.963768 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:35.964218 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:35.965019 17500 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:20:35.968518 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:20:35.968604 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 20:20:35.968680 17474 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:20:35.970155 17474 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:20:35.970224 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 20:20:35.970475 17500 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Bootstrap starting.
I20260620 20:20:35.981060 17474 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:37605
I20260620 20:20:35.981122 17607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:37605 every 8 connection(s)
I20260620 20:20:35.981930 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:35.983345 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17474
I20260620 20:20:35.983814 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:34429
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:43327
--webserver_port=36661
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:35.987656 17500 log.cc:824] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:20:35 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:35.995184 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.983736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:37479 (local address 127.2.235.254:43327)
0620 20:20:35.983889 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:35.983896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:35.984514 (+ 618us) server_negotiation.cc:407] Connection header received
0620 20:20:35.985710 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:35.985714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:35.985777 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:35.985856 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:35.986909 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.987462 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.988853 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.989180 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.989342 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.991643 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:35.991647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:35.991650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:35.991675 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:35.993114 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.993648 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.993654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.993656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.993740 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:35.994099 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:35.994102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:35.994104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:35.994275 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:35.994385 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:35.994858 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 20:20:35.994992 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:20:35.996027 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:35.983829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:37479)
0620 20:20:35.984358 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:35.984411 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:35.985510 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:35.985998 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:35.986005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:35.986308 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:35.986741 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.986754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.987622 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.987625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.988716 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:35.988726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:35.989332 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:35.989336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:35.989509 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:35.990089 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:35.990105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:35.991484 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:35.993273 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.993279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.993291 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.993544 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.993850 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:35.993853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:35.993855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:35.994000 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:35.994383 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:35.994389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:35.994618 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 20:20:35.994798 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 20:20:35.995094 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260620 20:20:35.996388 17500 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:35.996802 17500 tablet_bootstrap.cc:492] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Bootstrap complete.
I20260620 20:20:35.997175 17608 heartbeater.cc:347] Connected to a master server at 127.2.235.254:43327
I20260620 20:20:35.997421 17608 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:35.997910 17608 heartbeater.cc:511] Master 127.2.235.254:43327 requested a full tablet report, sending...
I20260620 20:20:35.998193 17500 ts_tablet_manager.cc:1397] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.008s
I20260620 20:20:35.999873 17281 ts_manager.cc:194] Registered new tserver with Master: 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605)
I20260620 20:20:36.001000 17281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:37479
I20260620 20:20:36.000782 17500 raft_consensus.cc:348] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:36.001224 17500 raft_consensus.cc:729] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75227bd7b5bf4129a64d8a30606b789e, State: Initialized, Role: FOLLOWER
I20260620 20:20:36.001713 17500 consensus_queue.cc:260] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } }
I20260620 20:20:36.002550 17500 ts_tablet_manager.cc:1428] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 20:20:36.003216 17608 heartbeater.cc:503] Master 127.2.235.254:43327 was elected leader, sending a full tablet report...
Jun 20 20:20:36 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986835, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.194@KRBTEST.COM
I20260620 20:20:36.010489 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.000674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:37605 (local address 127.0.0.1:33066)
0620 20:20:36.000915 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.000929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:36.001002 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:36.001937 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:36.001940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:36.002108 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:36.002201 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.002209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.003550 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.003555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.004439 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.004449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.005140 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.005144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.005237 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.005690 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:36.005702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:36.007004 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:36.008912 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:36.008916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:36.008918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:36.009119 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:36.009481 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:36.009485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:36.009487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:36.009575 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:36.009988 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:36.009991 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:36.010079 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:20:36.010194 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:20:36.010325 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 20:20:36.010550 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.001195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.2.235.194:37605)
0620 20:20:36.001502 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.001515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.001535 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:36.001591 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.001596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.001766 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.002373 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:36.002525 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.003433 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.004596 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.005026 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.005141 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.007139 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:36.007149 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:36.007160 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:36.007191 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:36.008777 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.009230 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.009236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.009237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.009378 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.009676 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.009681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.009682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.009883 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:36.010015 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.010269 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.010402 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
I20260620 20:20:36.011307 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet 3398ab79019046339ee63baa934ad072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:35 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33066
I20260620 20:20:36.012010 17615 tablet_replica.cc:330] stopping tablet replica
I20260620 20:20:36.012260 17615 raft_consensus.cc:2229] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:36.012455 17615 raft_consensus.cc:2258] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:36.013307 17615 ts_tablet_manager.cc:1905] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:36.017057 17615 ts_tablet_manager.cc:1918] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:36.017144 17615 log.cc:1196] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/3398ab79019046339ee63baa934ad072
I20260620 20:20:36.017424 17615 ts_tablet_manager.cc:1939] T 3398ab79019046339ee63baa934ad072 P 75227bd7b5bf4129a64d8a30606b789e: Deleting consensus metadata
I20260620 20:20:36.018829 17265 catalog_manager.cc:4841] TS 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605): tablet 3398ab79019046339ee63baa934ad072 (table test-table [id=6f92f936b50047cb8707db3e69f76c1e]) successfully deleted
W20260620 20:20:36.092767 17611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:36.092985 17611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:36.093050 17611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:36.096697 17611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:36.096772 17611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:36.096856 17611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:20:36.101657 17611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=36661
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:36.102461 17611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:36.103307 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:36.111420 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.111406 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.111506 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:36.111539 17611 server_base.cc:1034] running on GCE node
I20260620 20:20:36.111992 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:36.112550 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:36.113688 17611 hybrid_clock.cc:648] HybridClock initialized: now 1781986836113676 us; error 23 us; skew 500 ppm
Jun 20 20:20:36 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:36.116068 17611 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:20:36.116761 17611 webserver.cc:460] Webserver started at http://127.2.235.195:36661/ using document root <none> and password file <none>
I20260620 20:20:36.117225 17611 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:36.117357 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:36.120736 17611 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:20:36.122599 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.123636 17611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:20:36.123740 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "b0452d636287495281bfc12df81f04ed"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "b10b00b4528d18bc8f333f6bb8b534e4"
server_key_iv: "d762b9206cbff59c8693881d2bf32f15"
server_key_version: "encryptionkey@0"
I20260620 20:20:36.124189 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:36.147441 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:36.150358 17611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:36.150511 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:36.151029 17611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:36.151755 17611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:36.151794 17611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.151849 17611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:36.151878 17611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.160502 17611 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:34429
I20260620 20:20:36.160528 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:34429 every 8 connection(s)
I20260620 20:20:36.161491 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
Jun 20 20:20:36 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:36.171255 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17611
I20260620 20:20:36.173660 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.163260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:60517 (local address 127.2.235.254:43327)
0620 20:20:36.163412 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.163416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.164114 (+ 698us) server_negotiation.cc:407] Connection header received
0620 20:20:36.165175 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.165178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.165220 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.165376 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:36.166412 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.166929 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.168047 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.168231 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.168292 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.170437 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:36.170440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:36.170441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:36.170460 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:36.171849 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.172345 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.172358 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.172362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.172442 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.172742 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.172747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.172751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.172954 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:36.173046 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.173479 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.173559 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:20:36.174417 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.163587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:60517)
0620 20:20:36.163980 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.164018 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:36.165005 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:36.165539 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:36.165545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:36.165858 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:36.166257 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.166269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.167040 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.167044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.167929 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.167939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.168361 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.168364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.168490 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.168916 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:36.168931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:36.170293 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:36.171967 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:36.171974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:36.171988 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:36.172272 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:36.172518 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:36.172520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:36.172522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:36.172661 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:36.173047 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:36.173051 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:36.173244 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 20:20:36.173447 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 20:20:36.173698 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260620 20:20:36.175423 17744 heartbeater.cc:347] Connected to a master server at 127.2.235.254:43327
I20260620 20:20:36.175640 17744 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:36.176160 17744 heartbeater.cc:511] Master 127.2.235.254:43327 requested a full tablet report, sending...
I20260620 20:20:36.177392 17281 ts_manager.cc:194] Registered new tserver with Master: b0452d636287495281bfc12df81f04ed (127.2.235.195:34429)
I20260620 20:20:36.178180 17281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:60517
I20260620 20:20:36.179733 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.171896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.2.235.254:43327)
0620 20:20:36.172067 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.172074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.172267 (+ 193us) server_negotiation.cc:407] Connection header received
0620 20:20:36.172407 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.172410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.172442 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.172525 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:36.173273 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.173935 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.174808 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.175086 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.175185 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.176553 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:36.176556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:36.176557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:36.176576 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:36.178301 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.178722 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.178724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.178725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.178788 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:36.179064 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:36.179066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:36.179067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:36.179200 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:36.179342 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.179511 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.179591 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260620 20:20:36.181233 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:36.184335 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.181824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.2.235.254:43327)
0620 20:20:36.181918 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.181920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.182069 (+ 149us) server_negotiation.cc:407] Connection header received
0620 20:20:36.182156 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.182158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.182221 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.182292 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:36.182413 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.182800 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.183444 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.183614 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.183674 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.183859 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:36.184112 (+ 253us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:36.184183 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.184225 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.184262 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:20:36.184794 17281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:36.187654 17281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:36.193508 17542 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1ae2f7505a471f937f83a86455520d (DEFAULT_TABLE table=test-table [id=f9fa50b23ddc4b9baed9a5bf9e43e6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:36.193655 17403 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1ae2f7505a471f937f83a86455520d (DEFAULT_TABLE table=test-table [id=f9fa50b23ddc4b9baed9a5bf9e43e6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:36.194053 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1ae2f7505a471f937f83a86455520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:36.194154 17403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1ae2f7505a471f937f83a86455520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:36.196329 17500 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Bootstrap starting.
I20260620 20:20:36.196797 17361 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Bootstrap starting.
I20260620 20:20:36.197166 17500 tablet_bootstrap.cc:654] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:36.197678 17361 tablet_bootstrap.cc:654] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:36.197698 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.192810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:34429 (local address 127.0.0.1:49814)
0620 20:20:36.193078 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.193094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:36.193156 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:36.193756 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:36.193759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:36.193774 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:36.193845 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.193870 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.195183 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.195186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.196226 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.196236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.197313 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.197316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.197408 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.197445 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:20:36.197528 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:20:36.197594 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260620 20:20:36.197755 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.192903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.2.235.195:34429)
0620 20:20:36.193357 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.193367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.193386 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:20:36.193437 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.193442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.193590 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.193758 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:36.193952 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.194929 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.196458 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.197173 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.197311 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.197438 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.197556 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.197637 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":71,"threads_started":1}
I20260620 20:20:36.198868 17500 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: No bootstrap required, opened a new log
I20260620 20:20:36.198964 17500 ts_tablet_manager.cc:1397] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:36.199127 17361 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: No bootstrap required, opened a new log
I20260620 20:20:36.199199 17361 ts_tablet_manager.cc:1397] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:36.199457 17500 raft_consensus.cc:348] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.199572 17500 raft_consensus.cc:374] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:36.199611 17500 raft_consensus.cc:729] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75227bd7b5bf4129a64d8a30606b789e, State: Initialized, Role: FOLLOWER
I20260620 20:20:36.199620 17361 raft_consensus.cc:348] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.199728 17361 raft_consensus.cc:374] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:36.199769 17361 raft_consensus.cc:729] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb55acc50b8447884d17ff972549caa, State: Initialized, Role: FOLLOWER
I20260620 20:20:36.199736 17500 consensus_queue.cc:260] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.199908 17361 consensus_queue.cc:260] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.200042 17500 ts_tablet_manager.cc:1428] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:20:36.200174 17361 ts_tablet_manager.cc:1428] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:20:36.201226 17678 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1ae2f7505a471f937f83a86455520d (DEFAULT_TABLE table=test-table [id=f9fa50b23ddc4b9baed9a5bf9e43e6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:36.202062 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1ae2f7505a471f937f83a86455520d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:36.208055 17756 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Bootstrap starting.
I20260620 20:20:36.209990 17756 tablet_bootstrap.cc:654] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:36.210644 17756 log.cc:824] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:36.212363 17756 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: No bootstrap required, opened a new log
I20260620 20:20:36.212539 17756 ts_tablet_manager.cc:1397] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:20:36.215817 17756 raft_consensus.cc:348] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.216017 17756 raft_consensus.cc:374] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:36.216048 17756 raft_consensus.cc:729] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0452d636287495281bfc12df81f04ed, State: Initialized, Role: FOLLOWER
I20260620 20:20:36.216342 17756 consensus_queue.cc:260] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.216856 17744 heartbeater.cc:503] Master 127.2.235.254:43327 was elected leader, sending a full tablet report...
I20260620 20:20:36.217095 17756 ts_tablet_manager.cc:1428] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 20:20:36.232834 17609 tablet.cc:2369] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:20:36.289811 17470 tablet.cc:2369] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:36.306811 17758 raft_consensus.cc:482] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:36.306957 17758 raft_consensus.cc:504] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.307937 17758 leader_election.cc:288] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605), 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791)
I20260620 20:20:36.312742 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.308090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:37605 (local address 127.2.235.195:41787)
0620 20:20:36.308244 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.308256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:36.308359 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:36.308864 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:36.308867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:36.308886 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:36.308993 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.308998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.309993 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.309996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.311439 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.311450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.312330 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.312334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.312423 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.312444 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:20:36.312530 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:20:36.312607 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":125}
I20260620 20:20:36.312742 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.308493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:41787 (local address 127.2.235.194:37605)
0620 20:20:36.308608 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.308611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.308622 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:20:36.308665 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.308666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.308727 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.308832 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:36.309122 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.309879 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.311574 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.312207 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.312319 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.312442 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.312560 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.312644 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:20:36.313060 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.308259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:53715 (local address 127.2.235.193:35791)
0620 20:20:36.308400 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.308404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.308740 (+ 336us) server_negotiation.cc:407] Connection header received
0620 20:20:36.308906 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.308908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.308966 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.309030 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:36.309331 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.309986 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.311574 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.312207 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.312319 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.312442 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.312793 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.312933 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:20:36.313395 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.308139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:35791 (local address 127.2.235.195:53715)
0620 20:20:36.308645 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.308658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:36.308748 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:36.309063 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:36.309069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:36.309087 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:36.309193 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.309197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.310072 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.310075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.311440 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:36.311450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.312524 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.312527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:36.312645 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.312667 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:20:36.313178 (+ 511us) client_negotiation.cc:240] Negotiation successful
0620 20:20:36.313269 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":82,"spinlock_wait_cycles":120576,"thread_start_us":185,"threads_started":1}
I20260620 20:20:36.313390 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1ae2f7505a471f937f83a86455520d" candidate_uuid: "b0452d636287495281bfc12df81f04ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75227bd7b5bf4129a64d8a30606b789e" is_pre_election: true
I20260620 20:20:36.313702 17562 raft_consensus.cc:2454] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0452d636287495281bfc12df81f04ed in term 0.
I20260620 20:20:36.313875 17423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1ae2f7505a471f937f83a86455520d" candidate_uuid: "b0452d636287495281bfc12df81f04ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb55acc50b8447884d17ff972549caa" is_pre_election: true
I20260620 20:20:36.314111 17423 raft_consensus.cc:2454] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0452d636287495281bfc12df81f04ed in term 0.
I20260620 20:20:36.314175 17631 leader_election.cc:302] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed; no voters:
I20260620 20:20:36.314389 17758 raft_consensus.cc:2781] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:36.314464 17758 raft_consensus.cc:482] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:36.314515 17758 raft_consensus.cc:3037] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:36.315411 17758 raft_consensus.cc:504] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.315781 17758 leader_election.cc:288] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [CANDIDATE]: Term 1 election: Requested vote from peers 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605), 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791)
I20260620 20:20:36.316095 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1ae2f7505a471f937f83a86455520d" candidate_uuid: "b0452d636287495281bfc12df81f04ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75227bd7b5bf4129a64d8a30606b789e"
I20260620 20:20:36.316179 17423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1ae2f7505a471f937f83a86455520d" candidate_uuid: "b0452d636287495281bfc12df81f04ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb55acc50b8447884d17ff972549caa"
I20260620 20:20:36.316237 17562 raft_consensus.cc:3037] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:36.316336 17423 raft_consensus.cc:3037] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:36.317339 17562 raft_consensus.cc:2454] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0452d636287495281bfc12df81f04ed in term 1.
I20260620 20:20:36.317463 17423 raft_consensus.cc:2454] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0452d636287495281bfc12df81f04ed in term 1.
I20260620 20:20:36.317597 17631 leader_election.cc:302] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed; no voters:
I20260620 20:20:36.317786 17758 raft_consensus.cc:2781] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:36.318023 17758 raft_consensus.cc:686] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 1 LEADER]: Becoming Leader. State: Replica: b0452d636287495281bfc12df81f04ed, State: Running, Role: LEADER
I20260620 20:20:36.318303 17758 consensus_queue.cc:237] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.321180 17280 catalog_manager.cc:5495] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed reported cstate change: term changed from 0 to 1, leader changed from <none> to b0452d636287495281bfc12df81f04ed (127.2.235.195). New cstate: current_term: 1 leader_uuid: "b0452d636287495281bfc12df81f04ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } health_report { overall_health: UNKNOWN } } }
I20260620 20:20:36.349403 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.346268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.2.235.195:34429)
0620 20:20:36.346393 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:36.346396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:36.346409 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:20:36.346504 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:36.346506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:36.346551 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:36.346636 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:36.346848 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.347314 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.348125 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:36.348342 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:36.348415 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:36.348640 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:36.349025 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:36.349111 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:36.349159 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 20:20:36.349201 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:20:36.354394 17562 raft_consensus.cc:1261] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Refusing update from remote peer b0452d636287495281bfc12df81f04ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:36.354398 17423 raft_consensus.cc:1261] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Refusing update from remote peer b0452d636287495281bfc12df81f04ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:36.354954 17760 consensus_queue.cc:1035] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:36.355065 17758 consensus_queue.cc:1035] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:36.361299 17764 mvcc.cc:205] Tried to move back new op lower bound from 7299018081701376000 to 7299018081562279936. Current Snapshot: MvccSnapshot[applied={T|T < 7299018081701376000}]
I20260620 20:20:36.362870 17763 mvcc.cc:205] Tried to move back new op lower bound from 7299018081701376000 to 7299018081562279936. Current Snapshot: MvccSnapshot[applied={T|T < 7299018081701376000 or (T in {7299018081701376000})}]
I20260620 20:20:36.367416 17280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:36.367530 17280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:36.369462 17280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Sending DeleteTablet for 3 replicas of tablet 9c1ae2f7505a471f937f83a86455520d
I20260620 20:20:36.370054 17678 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1ae2f7505a471f937f83a86455520d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:36 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:49814
I20260620 20:20:36.370082 17403 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1ae2f7505a471f937f83a86455520d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:36 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:43468
I20260620 20:20:36.370143 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 17319
I20260620 20:20:36.370409 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c1ae2f7505a471f937f83a86455520d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:36 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:33066
I20260620 20:20:36.370566 17615 tablet_replica.cc:330] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: stopping tablet replica
I20260620 20:20:36.370673 17615 raft_consensus.cc:2229] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:36.370795 17615 raft_consensus.cc:2258] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:36.371434 17615 ts_tablet_manager.cc:1905] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:36.373457 17773 tablet_replica.cc:330] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: stopping tablet replica
I20260620 20:20:36.373711 17773 raft_consensus.cc:2229] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:36.373836 17615 ts_tablet_manager.cc:1918] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:36.373932 17615 log.cc:1196] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/9c1ae2f7505a471f937f83a86455520d
I20260620 20:20:36.374006 17773 raft_consensus.cc:2258] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:36.374187 17615 ts_tablet_manager.cc:1939] T 9c1ae2f7505a471f937f83a86455520d P 75227bd7b5bf4129a64d8a30606b789e: Deleting consensus metadata
I20260620 20:20:36.374806 17265 catalog_manager.cc:4841] TS 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605): tablet 9c1ae2f7505a471f937f83a86455520d (table test-table [id=f9fa50b23ddc4b9baed9a5bf9e43e6d7]) successfully deleted
I20260620 20:20:36.375339 17773 ts_tablet_manager.cc:1905] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:20:36.377722 17266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:35791 (error 108)
I20260620 20:20:36.378412 17773 ts_tablet_manager.cc:1918] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:36.378499 17773 log.cc:1196] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/9c1ae2f7505a471f937f83a86455520d
W20260620 20:20:36.378811 17476 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:20:36.378472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:35791 (local address 127.0.0.1:43482)
0620 20:20:36.378649 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.378722 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:35791: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 20:20:36.378829 17773 ts_tablet_manager.cc:1939] T 9c1ae2f7505a471f937f83a86455520d P b0452d636287495281bfc12df81f04ed: Deleting consensus metadata
I20260620 20:20:36.378820 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 17474
I20260620 20:20:36.379844 17265 catalog_manager.cc:4841] TS b0452d636287495281bfc12df81f04ed (127.2.235.195:34429): tablet 9c1ae2f7505a471f937f83a86455520d (table test-table [id=f9fa50b23ddc4b9baed9a5bf9e43e6d7]) successfully deleted
W20260620 20:20:36.380632 17266 catalog_manager.cc:4584] TS 4eb55acc50b8447884d17ff972549caa (127.2.235.193:35791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c1ae2f7505a471f937f83a86455520d: Network error: Client connection negotiation failed: client connection to 127.2.235.193:35791: connect: Connection reset by peer (error 104)
I20260620 20:20:36.384579 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 17611
I20260620 20:20:36.389905 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 17248
I20260620 20:20:36.395624 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43327
--webserver_interface=127.2.235.254
--webserver_port=39821
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:36.494668 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:36.494851 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:36.494884 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:36.498555 17775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:36.498598 17775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:36.498616 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:36.498631 17775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:36.498644 17775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:36.503389 17775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=39821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:36.504189 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:36.505028 17775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:36.512194 17780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.512212 17781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:36.512400 17775 server_base.cc:1034] running on GCE node
W20260620 20:20:36.512174 17783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:36.512995 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:36.513942 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:36.515090 17775 hybrid_clock.cc:648] HybridClock initialized: now 1781986836515077 us; error 24 us; skew 500 ppm
Jun 20 20:20:36 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:36.517530 17775 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:36.518273 17775 webserver.cc:460] Webserver started at http://127.2.235.254:39821/ using document root <none> and password file <none>
I20260620 20:20:36.518682 17775 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:36.518731 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:36.521895 17775 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:20:36.523761 17790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.524844 17775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:36.524966 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "8881d861b0ed4472b719208f0ee7d7a2"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "5761d06d3eb481a4d042cf4955c01f24"
server_key_iv: "454f71e2d1a93a0f8485b403928116f0"
server_key_version: "encryptionkey@0"
I20260620 20:20:36.525439 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:36.543255 17775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:36.546056 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:36.546217 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:36.552471 17775 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:43327
I20260620 20:20:36.552735 17842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:43327 every 8 connection(s)
I20260620 20:20:36.553444 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:36.557088 17843 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:20:36.558593 17843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap starting.
I20260620 20:20:36.560729 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17775
I20260620 20:20:36.561100 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:35791
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:43327
--webserver_port=41949
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:36.574649 17843 log.cc:824] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:36.585070 17843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:36.585443 17843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2: Bootstrap complete.
I20260620 20:20:36.589196 17843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:36.589628 17843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8881d861b0ed4472b719208f0ee7d7a2, State: Initialized, Role: FOLLOWER
I20260620 20:20:36.590060 17843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:36.590204 17843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:36.590277 17843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:36.590391 17843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:20:36.592139 17843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:36.592451 17843 leader_election.cc:302] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8881d861b0ed4472b719208f0ee7d7a2; no voters:
I20260620 20:20:36.592746 17843 leader_election.cc:288] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:20:36.592816 17847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:20:36.593005 17847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [term 3 LEADER]: Becoming Leader. State: Replica: 8881d861b0ed4472b719208f0ee7d7a2, State: Running, Role: LEADER
I20260620 20:20:36.593291 17847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } }
I20260620 20:20:36.593909 17843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:36.594158 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8881d861b0ed4472b719208f0ee7d7a2. Latest consensus state: current_term: 3 leader_uuid: "8881d861b0ed4472b719208f0ee7d7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } } }
I20260620 20:20:36.594273 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:36.594602 17851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:36.594722 17848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8881d861b0ed4472b719208f0ee7d7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8881d861b0ed4472b719208f0ee7d7a2" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43327 } } }
I20260620 20:20:36.594882 17848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8881d861b0ed4472b719208f0ee7d7a2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:36.599670 17851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:36.600340 17851 catalog_manager.cc:1259] Loaded cluster ID: 8464e37fb7324afda6a8aaa0bdebc001
I20260620 20:20:36.600409 17851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 20:20:36.601179 17851 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9f800369ad google::LogMessage::Fail() at ??:0
@ 0x7f9f8003988a google::LogMessage::SendToLog() at ??:0
@ 0x7f9f800364de google::LogMessage::Flush() at ??:0
@ 0x7f9f80037a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9f8a975fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9f8a976d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9f8a973579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9f8a9b7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9f822883d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9f82288c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9f8228a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9f82279512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9f838bd6db start_thread at ??:0
@ 0x7f9f7eb3461f clone at ??:0
W20260620 20:20:36.692272 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:36.692479 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:36.692541 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:36.696297 17845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:36.696353 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:36.696462 17845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:20:36.701411 17845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:35791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=41949
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:36.702261 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:36.703130 17845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:36.711956 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.712042 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.711956 17885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:36.712327 17845 server_base.cc:1034] running on GCE node
I20260620 20:20:36.712672 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:36.713212 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:36.714411 17845 hybrid_clock.cc:648] HybridClock initialized: now 1781986836714383 us; error 40 us; skew 500 ppm
Jun 20 20:20:36 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:36.716993 17845 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:20:36.717854 17845 webserver.cc:460] Webserver started at http://127.2.235.193:41949/ using document root <none> and password file <none>
I20260620 20:20:36.718357 17845 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:36.718417 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:36.722223 17845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:20:36.724342 17894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.725571 17845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:36.725688 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "4eb55acc50b8447884d17ff972549caa"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "605ff08ddaa891fa1d72b49679fd6cd2"
server_key_iv: "38e1ae9b13ba82c79a1ef7c8317bf48a"
server_key_version: "encryptionkey@0"
I20260620 20:20:36.726166 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:36.738498 17845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:36.741113 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:36.741339 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:36.741838 17845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:36.742780 17901 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:20:36.747573 17845 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:20:36.747641 17845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 20:20:36.747712 17845 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:20:36.749150 17845 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:20:36.749213 17845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 20:20:36.749506 17901 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Bootstrap starting.
I20260620 20:20:36.759436 17845 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:35791
I20260620 20:20:36.759665 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:35791 every 8 connection(s)
I20260620 20:20:36.760334 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:36.766837 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 17845
I20260620 20:20:36.767244 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:37605
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:43327
--webserver_port=45637
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:20:36.770077 17901 log.cc:824] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:36.778407 17901 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:20:36.778775 17901 tablet_bootstrap.cc:492] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Bootstrap complete.
I20260620 20:20:36.780135 17901 ts_tablet_manager.cc:1397] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.007s
I20260620 20:20:36.783133 17901 raft_consensus.cc:348] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.783627 17901 raft_consensus.cc:729] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb55acc50b8447884d17ff972549caa, State: Initialized, Role: FOLLOWER
I20260620 20:20:36.784114 17901 consensus_queue.cc:260] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:36.785120 17901 ts_tablet_manager.cc:1428] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260620 20:20:36.791916 18011 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:20:36.762120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:35463)
0620 20:20:36.762565 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.762613 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:36.763312 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:36.791211 (+ 27899us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
W20260620 20:20:36.792315 18009 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:43327 (0 consecutive failures): Network error: Failed to ping master at 127.2.235.254:43327: Client connection negotiation failed: client connection to 127.2.235.254:43327: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:20:36.792915 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.792675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:36265)
0620 20:20:36.792786 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.792865 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 20:20:36.793529 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.793332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:58985)
0620 20:20:36.793437 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.793479 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 20:20:36.793675 18009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:20:36.872720 18012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:36.872947 18012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:36.873014 18012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:36.876606 18012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:36.876682 18012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:36.876792 18012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:20:36.881614 18012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:37605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=45637
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:36.882481 18012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:36.883365 18012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:36.891973 18020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.891969 18019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:36.892040 18022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:36.892155 18012 server_base.cc:1034] running on GCE node
I20260620 20:20:36.892659 18012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:36.893169 18012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:36.894348 18012 hybrid_clock.cc:648] HybridClock initialized: now 1781986836894333 us; error 27 us; skew 500 ppm
Jun 20 20:20:36 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:36.896724 18012 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:20:36.897607 18012 webserver.cc:460] Webserver started at http://127.2.235.194:45637/ using document root <none> and password file <none>
I20260620 20:20:36.898083 18012 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:36.898134 18012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:36.901393 18012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:20:36.903210 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.904309 18012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:36.904393 18012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "75227bd7b5bf4129a64d8a30606b789e"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "4e38a656f3eedfbed589c5074b7ca7ff"
server_key_iv: "c817c4a60b9377bd92a3588e61229113"
server_key_version: "encryptionkey@0"
I20260620 20:20:36.904798 18012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:36.923127 18012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:36.925637 18012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:36.925768 18012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:36.926219 18012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:36.926924 18012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:36.926970 18012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.927006 18012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:36.927026 18012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:36.935495 18012 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:37605
I20260620 20:20:36.935521 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:37605 every 8 connection(s)
I20260620 20:20:36.936368 18012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:36.939774 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.938227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:34027)
0620 20:20:36.938625 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.938717 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":110,"threads_started":1}
W20260620 20:20:36.940200 18143 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:43327 (0 consecutive failures): Network error: Failed to ping master at 127.2.235.254:43327: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
I20260620 20:20:36.940762 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.940574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:49219)
0620 20:20:36.940669 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.940713 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 20:20:36.941429 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:36.941222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:38257)
0620 20:20:36.941329 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:36.941381 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:20:36.941581 18143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:20:36.943819 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18012
I20260620 20:20:36.944249 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:34429
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:43327
--webserver_port=36661
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:37.042510 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:37.042721 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:37.042758 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:37.046427 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:37.046468 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:37.046555 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:20:37.051229 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:34429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=36661
--tserver_master_addrs=127.2.235.254:43327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:37.052029 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:37.052860 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:37.060997 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:37.061048 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:37.061182 18146 server_base.cc:1034] running on GCE node
W20260620 20:20:37.061008 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:37.061656 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:37.062224 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:37.063398 18146 hybrid_clock.cc:648] HybridClock initialized: now 1781986837063369 us; error 42 us; skew 500 ppm
Jun 20 20:20:37 dist-test-slave-8z53 krb5kdc[16696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986837, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:37.065949 18146 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:20:37.066723 18146 webserver.cc:460] Webserver started at http://127.2.235.195:36661/ using document root <none> and password file <none>
I20260620 20:20:37.067186 18146 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:37.067246 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:37.070703 18146 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 20:20:37.072496 18161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:37.073590 18146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:20:37.073706 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "b0452d636287495281bfc12df81f04ed"
format_stamp: "Formatted at 2026-06-20 20:20:34 on dist-test-slave-8z53"
server_key: "b10b00b4528d18bc8f333f6bb8b534e4"
server_key_iv: "d762b9206cbff59c8693881d2bf32f15"
server_key_version: "encryptionkey@0"
I20260620 20:20:37.074160 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:37.108634 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:37.111267 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:37.111441 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:37.111964 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:37.112669 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:37.112735 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:37.112799 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:37.112870 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:37.121443 18274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:34429 every 8 connection(s)
I20260620 20:20:37.121457 18146 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:34429
I20260620 20:20:37.122334 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:37.125622 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:37.124075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:52891)
0620 20:20:37.124514 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:37.124916 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":94,"threads_started":1}
W20260620 20:20:37.126024 18275 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:43327 (0 consecutive failures): Network error: Failed to ping master at 127.2.235.254:43327: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
I20260620 20:20:37.126489 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:37.126303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:45977)
0620 20:20:37.126393 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:37.126438 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 20:20:37.127130 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:37.126942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:46573)
0620 20:20:37.127034 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:37.127079 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 20:20:37.127332 18275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:20:37.129889 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18146
I20260620 20:20:37.794461 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:37.794107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:34651)
0620 20:20:37.794325 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:37.794378 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":57,"threads_started":1}
I20260620 20:20:37.942365 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:37.942038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:41711)
0620 20:20:37.942230 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:37.942275 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260620 20:20:38.128206 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.127841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:47177)
0620 20:20:38.128085 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:38.128126 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260620 20:20:38.654335 18282 raft_consensus.cc:482] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:38.654547 18282 raft_consensus.cc:504] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:38.655617 18282 leader_election.cc:288] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0452d636287495281bfc12df81f04ed (127.2.235.195:34429), 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605)
Jun 20 20:20:38 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.195@KRBTEST.COM
Jun 20 20:20:38 dist-test-slave-8z53 krb5kdc[16696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986836, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.194@KRBTEST.COM
I20260620 20:20:38.669215 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.656069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50619 (local address 127.2.235.195:34429)
0620 20:20:38.656481 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:38.656496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:38.656541 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:20:38.657379 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:38.657388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:38.657780 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:38.658080 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:38.659252 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.660297 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.661623 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.662066 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.662225 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.664662 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:38.664668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:38.664682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:38.664715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:38.666538 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.667204 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.667209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.667212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.667310 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.667900 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.667903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.667905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.668358 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:38.668448 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:38.668888 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 20:20:38.669082 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":83,"threads_started":1}
I20260620 20:20:38.669380 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.655805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:34429 (local address 127.2.235.193:50619)
0620 20:20:38.655997 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:38.656010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:38.656107 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:38.658133 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:38.658138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:38.658667 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:38.659106 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:38.659119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.660540 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.660545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:38.661496 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:38.661505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.662177 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.662180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:38.662338 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.663016 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:38.663031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:38.664450 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:38.666657 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:38.666662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:38.666674 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:38.667085 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:38.667405 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:38.667409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:38.667411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:38.667558 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:38.668444 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:38.668447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:38.668579 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 20:20:38.668787 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 20:20:38.669123 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":153,"spinlock_wait_cycles":21248,"thread_start_us":73,"threads_started":1}
I20260620 20:20:38.669921 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.656301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:34241 (local address 127.2.235.194:37605)
0620 20:20:38.656622 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:38.656641 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:38.656671 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:20:38.657483 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:38.657495 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:38.657826 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:38.658116 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:38.659255 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.660264 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.661629 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.662094 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.662264 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.664955 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:38.664959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:38.664969 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:38.664995 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:38.666692 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.667176 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.667184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.667192 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.667302 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:38.667642 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:38.667645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:38.667647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:38.668155 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:38.668272 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:38.669560 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0620 20:20:38.669787 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260620 20:20:38.669379 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.656131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:37605 (local address 127.2.235.193:34241)
0620 20:20:38.656402 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:38.656414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:38.656532 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:38.658132 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:38.658137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:38.658666 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:38.659106 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:38.659120 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.660465 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.660469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:38.661501 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:38.661511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:38.662219 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:38.662223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:38.662338 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:38.663079 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:38.663092 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:38.664786 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:38.666799 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:38.666802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:38.666803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:38.667075 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:38.667405 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:38.667408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:38.667411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:38.667554 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:38.668332 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:38.668336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:38.668579 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 20:20:38.668787 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 20:20:38.669175 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":304,"spinlock_wait_cycles":30976,"thread_start_us":56,"threads_started":1}
W20260620 20:20:38.671666 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0452d636287495281bfc12df81f04ed (127.2.235.195:34429): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
W20260620 20:20:38.671859 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
I20260620 20:20:38.671931 17895 leader_election.cc:302] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb55acc50b8447884d17ff972549caa; no voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed
I20260620 20:20:38.672159 18282 raft_consensus.cc:2726] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:38.795308 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.795087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:40233)
0620 20:20:38.795192 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:38.795237 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 20:20:38.943346 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:38.942993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:39675)
0620 20:20:38.943203 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:38.943249 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260620 20:20:39.129292 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:39.128839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:35435)
0620 20:20:39.129047 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:39.129097 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260620 20:20:39.796373 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:39.795998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:41175)
0620 20:20:39.796230 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:39.796279 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260620 20:20:39.944344 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:39.944003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:55327)
0620 20:20:39.944221 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:39.944267 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260620 20:20:40.130362 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.129988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:34229)
0620 20:20:40.130201 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.130258 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260620 20:20:40.647365 18292 raft_consensus.cc:482] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:40.647502 18292 raft_consensus.cc:504] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:40.647969 18292 leader_election.cc:288] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0452d636287495281bfc12df81f04ed (127.2.235.195:34429), 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605)
W20260620 20:20:40.648785 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0452d636287495281bfc12df81f04ed (127.2.235.195:34429): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
W20260620 20:20:40.648970 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
I20260620 20:20:40.649034 17895 leader_election.cc:302] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb55acc50b8447884d17ff972549caa; no voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed
I20260620 20:20:40.649192 18292 raft_consensus.cc:2726] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:40.797412 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.796991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:51097)
0620 20:20:40.797214 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.797298 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260620 20:20:40.945398 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:40.944946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:46221)
0620 20:20:40.945216 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:40.945315 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260620 20:20:41.131322 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:41.130943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:45481)
0620 20:20:41.131175 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:41.131232 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
I20260620 20:20:41.798509 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:41.798117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:33323)
0620 20:20:41.798370 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:41.798429 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
I20260620 20:20:41.946317 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:41.945988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:41117)
0620 20:20:41.946190 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:41.946239 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260620 20:20:42.132303 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.131970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:57749)
0620 20:20:42.132178 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:42.132226 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":60,"threads_started":1}
I20260620 20:20:42.799556 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.799209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:33317)
0620 20:20:42.799419 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:42.799468 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260620 20:20:42.947247 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:42.946904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:55459)
0620 20:20:42.947121 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:42.947168 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260620 20:20:43.133425 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.132989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:40449)
0620 20:20:43.133282 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.133346 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260620 20:20:43.585529 18302 raft_consensus.cc:482] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:43.585645 18302 raft_consensus.cc:504] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:43.586053 18302 leader_election.cc:288] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0452d636287495281bfc12df81f04ed (127.2.235.195:34429), 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605)
W20260620 20:20:43.586875 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
W20260620 20:20:43.587018 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0452d636287495281bfc12df81f04ed (127.2.235.195:34429): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
I20260620 20:20:43.587076 17895 leader_election.cc:302] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb55acc50b8447884d17ff972549caa; no voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed
I20260620 20:20:43.587244 18302 raft_consensus.cc:2726] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:43.800575 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.800201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:43699)
0620 20:20:43.800438 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.800487 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260620 20:20:43.948261 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:43.947872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:49211)
0620 20:20:43.948128 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:43.948178 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":65,"threads_started":1}
I20260620 20:20:44.134424 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:44.134023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:49171)
0620 20:20:44.134295 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:44.134346 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260620 20:20:44.801640 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:44.801168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:34661)
0620 20:20:44.801511 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:44.801556 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":98,"threads_started":1}
I20260620 20:20:44.949398 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:44.948918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:46041)
0620 20:20:44.949200 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:44.949314 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":66,"threads_started":1}
I20260620 20:20:45.135380 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:45.135052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:59057)
0620 20:20:45.135254 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:45.135299 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260620 20:20:45.802724 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:45.802328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:53493)
0620 20:20:45.802572 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:45.802622 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260620 20:20:45.950410 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:45.950021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:50207)
0620 20:20:45.950267 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:45.950323 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
I20260620 20:20:46.136343 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:46.135982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:45607)
0620 20:20:46.136204 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:46.136251 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260620 20:20:46.803808 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:46.803425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:57517)
0620 20:20:46.803671 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:46.803721 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":61,"threads_started":1}
I20260620 20:20:46.951424 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:46.951070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:39439)
0620 20:20:46.951287 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:46.951338 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260620 20:20:47.137338 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.136938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:40853)
0620 20:20:47.137144 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.137196 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":56,"threads_started":1}
I20260620 20:20:47.352334 18315 raft_consensus.cc:482] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:47.352452 18315 raft_consensus.cc:504] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0452d636287495281bfc12df81f04ed" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 34429 } } peers { permanent_uuid: "75227bd7b5bf4129a64d8a30606b789e" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 37605 } } peers { permanent_uuid: "4eb55acc50b8447884d17ff972549caa" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 35791 } }
I20260620 20:20:47.352833 18315 leader_election.cc:288] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0452d636287495281bfc12df81f04ed (127.2.235.195:34429), 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605)
W20260620 20:20:47.353595 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0452d636287495281bfc12df81f04ed (127.2.235.195:34429): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
W20260620 20:20:47.353770 17895 leader_election.cc:341] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75227bd7b5bf4129a64d8a30606b789e (127.2.235.194:37605): Not found: Tablet not found: 9c1ae2f7505a471f937f83a86455520d
I20260620 20:20:47.353832 17895 leader_election.cc:302] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb55acc50b8447884d17ff972549caa; no voters: 75227bd7b5bf4129a64d8a30606b789e, b0452d636287495281bfc12df81f04ed
I20260620 20:20:47.353971 18315 raft_consensus.cc:2726] T 9c1ae2f7505a471f937f83a86455520d P 4eb55acc50b8447884d17ff972549caa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:20:47.804900 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.804545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:60017)
0620 20:20:47.804763 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.804809 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260620 20:20:47.952435 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:47.952061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:44825)
0620 20:20:47.952280 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:47.952346 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260620 20:20:48.138358 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.137998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:44825)
0620 20:20:48.138223 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.138272 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260620 20:20:48.805949 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.805594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:41541)
0620 20:20:48.805814 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.805862 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
I20260620 20:20:48.953541 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:48.953148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:42911)
0620 20:20:48.953402 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:48.953453 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":64,"threads_started":1}
I20260620 20:20:49.139353 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:49.138995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:55983)
0620 20:20:49.139219 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:49.139265 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260620 20:20:49.807174 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:49.806842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:57875)
0620 20:20:49.807050 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:49.807095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260620 20:20:49.954492 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:49.954163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:45991)
0620 20:20:49.954366 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:49.954414 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260620 20:20:50.140367 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.139988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:33357)
0620 20:20:50.140232 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.140278 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260620 20:20:50.808257 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.807866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:45757)
0620 20:20:50.808119 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.808169 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
I20260620 20:20:50.955417 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:50.955079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:33189)
0620 20:20:50.955294 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:50.955340 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260620 20:20:51.141386 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.140980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:36493)
0620 20:20:51.141199 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.141242 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260620 20:20:51.809540 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.809097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.193:51987)
0620 20:20:51.809401 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.809451 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
I20260620 20:20:51.956487 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:51.956129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.194:50473)
0620 20:20:51.956358 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:51.956404 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260620 20:20:52.140542 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 17845
I20260620 20:20:52.142521 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:52.142148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:43327 (local address 127.2.235.195:36463)
0620 20:20:52.142373 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:52.142433 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260620 20:20:52.146166 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18012
I20260620 20:20:52.151345 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18146
2026-06-20T20:20:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19066 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:20:52 dist-test-slave-8z53 krb5kdc[18333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:20:52 dist-test-slave-8z53 krb5kdc[18333](info): set up 2 sockets
Jun 20 20:20:52 dist-test-slave-8z53 krb5kdc[18333](info): commencing operation
krb5kdc: starting...
W20260620 20:20:53.125882 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:20:53.226152 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42387
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:53.326519 18346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:53.326704 18346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:53.326735 18346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:53.330361 18346 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:20:53.330399 18346 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:20:53.330413 18346 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:20:53.330427 18346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:20:53.330444 18346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:53.330461 18346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:53.330492 18346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:20:53.330507 18346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:20:53.335031 18346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:53.335779 18346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:53.336572 18346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:53.344331 18354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.344341 18351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.344357 18346 server_base.cc:1034] running on GCE node
W20260620 20:20:53.344341 18352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.345129 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:53.346048 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:53.347247 18346 hybrid_clock.cc:648] HybridClock initialized: now 1781986853347229 us; error 28 us; skew 500 ppm
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:53.349766 18346 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:20:53.350489 18346 webserver.cc:460] Webserver started at http://127.2.235.254:43017/ using document root <none> and password file <none>
I20260620 20:20:53.350963 18346 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:53.351042 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:53.351236 18346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:53.353009 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "1e567951b53b4219928daf572f461847"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "985def8a1e700a752bf126e529a69bfc"
server_key_iv: "d113450d65c1371476b5b3233c8401b7"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.353545 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "1e567951b53b4219928daf572f461847"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "985def8a1e700a752bf126e529a69bfc"
server_key_iv: "d113450d65c1371476b5b3233c8401b7"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.356652 18346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:20:53.358923 18361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.360088 18346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:20:53.360232 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "1e567951b53b4219928daf572f461847"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "985def8a1e700a752bf126e529a69bfc"
server_key_iv: "d113450d65c1371476b5b3233c8401b7"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.360348 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:53.374411 18346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:53.376912 18346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:53.377115 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:53.383353 18346 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:42387
I20260620 20:20:53.383613 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:42387 every 8 connection(s)
I20260620 20:20:53.384183 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:20:53.386557 18414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:53.390756 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18346
I20260620 20:20:53.390895 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:20:53.391153 2991 external_mini_cluster.cc:1428] Setting key b277c5a0345a205f01db0ccf038cb1d6
I20260620 20:20:53.391676 18414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap starting.
I20260620 20:20:53.394042 18414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:53.394840 18414 log.cc:824] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:53.396699 18414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: No bootstrap required, opened a new log
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:53.400184 18414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:20:53.400421 18414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:53.400480 18414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e567951b53b4219928daf572f461847, State: Initialized, Role: FOLLOWER
I20260620 20:20:53.400935 18414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:20:53.401096 18414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:20:53.401171 18414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:20:53.401289 18414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:53.402004 18414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:20:53.402287 18414 leader_election.cc:302] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e567951b53b4219928daf572f461847; no voters:
I20260620 20:20:53.402576 18414 leader_election.cc:288] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:20:53.402630 18419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:53.402850 18419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 LEADER]: Becoming Leader. State: Replica: 1e567951b53b4219928daf572f461847, State: Running, Role: LEADER
I20260620 20:20:53.403156 18419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:20:53.403698 18414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:20:53.404613 18420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e567951b53b4219928daf572f461847. Latest consensus state: current_term: 1 leader_uuid: "1e567951b53b4219928daf572f461847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } } }
I20260620 20:20:53.405000 18420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:53.404888 18421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e567951b53b4219928daf572f461847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } } }
I20260620 20:20:53.405347 18421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: This master's current role is: LEADER
I20260620 20:20:53.405570 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.392426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.2.235.254:42387)
0620 20:20:53.392826 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:53.392843 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:53.392880 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:20:53.393631 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:53.393651 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:53.394029 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:53.394367 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:53.395019 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.396309 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.397301 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.397935 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.398083 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.400424 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:53.400432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:53.400449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:53.400476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:53.402157 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.402577 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.402583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.402588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.402741 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.403028 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.403033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.403035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.403472 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:53.403669 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:53.404277 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 20:20:53.404597 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
W20260620 20:20:53.408362 18434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:20:53.408449 18434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:20:53.408524 18429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:20:53.409190 18429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:20:53.414075 18429 catalog_manager.cc:1347] Generated new cluster ID: 833b3666744645a38fe6430578d96809
I20260620 20:20:53.414131 18429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:20:53.423167 18429 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:20:53.423962 18429 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:20:53.430899 18429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Generated new TSK 0
I20260620 20:20:53.431448 18429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:20:53.461458 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42387
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:53.559125 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:53.559321 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:53.559399 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:53.562983 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:53.563024 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:53.563109 18440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:20:53.567766 18440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:53.568607 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:53.569552 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:53.578006 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.578011 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.578136 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.578197 18440 server_base.cc:1034] running on GCE node
I20260620 20:20:53.578658 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:53.579247 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:53.580399 18440 hybrid_clock.cc:648] HybridClock initialized: now 1781986853580395 us; error 28 us; skew 500 ppm
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:53.582799 18440 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:20:53.583592 18440 webserver.cc:460] Webserver started at http://127.2.235.193:45559/ using document root <none> and password file <none>
I20260620 20:20:53.584060 18440 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:53.584138 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:53.584333 18440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:53.586122 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "daeb5f1388e741248570d0a0ceeab6a2"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "38febb8697d45d2f0743d8e493d3a75c"
server_key_iv: "5de8419c6f691847ea178f40dde3537a"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.586699 18440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "daeb5f1388e741248570d0a0ceeab6a2"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "38febb8697d45d2f0743d8e493d3a75c"
server_key_iv: "5de8419c6f691847ea178f40dde3537a"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.589959 18440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:53.591933 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.592890 18440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:53.593003 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "daeb5f1388e741248570d0a0ceeab6a2"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "38febb8697d45d2f0743d8e493d3a75c"
server_key_iv: "5de8419c6f691847ea178f40dde3537a"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.593096 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:53.606654 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:53.609208 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:53.609413 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:53.609925 18440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:53.610670 18440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:53.610744 18440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.610806 18440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:53.610849 18440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.619055 18440 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:38677
I20260620 20:20:53.619078 18568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:38677 every 8 connection(s)
I20260620 20:20:53.619879 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:20:53.626156 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18440
I20260620 20:20:53.626298 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:20:53.626566 2991 external_mini_cluster.cc:1428] Setting key 12d491acbdfe77052d69f2ceb9f98d76
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:53.632949 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.621514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50823 (local address 127.2.235.254:42387)
0620 20:20:53.621653 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:53.621658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:53.622628 (+ 970us) server_negotiation.cc:407] Connection header received
0620 20:20:53.623346 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:53.623349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:53.623390 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:53.623460 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:53.624483 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.625035 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.626219 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.626515 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.626621 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.629332 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:53.629339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:53.629345 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:53.629382 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:53.630871 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.631447 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.631450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.631451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.631516 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.631866 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.631868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.631869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.632107 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:53.632181 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:53.632745 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 20:20:53.632844 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:20:53.634075 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.621876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:50823)
0620 20:20:53.622477 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:53.622513 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:53.623185 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:53.623603 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:53.623609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:53.623925 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:53.624317 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:53.624328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.625154 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.625156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:53.626102 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:53.626110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.626650 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.626653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:53.626820 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.627406 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:53.627430 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:53.629149 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:53.630980 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:53.630989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:53.631004 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:53.631347 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:53.631604 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:53.631609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:53.631614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:53.631780 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:53.632213 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:53.632221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:53.632510 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 20:20:53.632713 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 20:20:53.633068 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":99,"threads_started":1}
I20260620 20:20:53.635326 18569 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42387
I20260620 20:20:53.635591 18569 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:53.636118 18569 heartbeater.cc:511] Master 127.2.235.254:42387 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:20:53.637722 18378 ts_manager.cc:194] Registered new tserver with Master: daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677)
I20260620 20:20:53.638737 18378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:50823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:20:53.651453 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42387
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:53.747843 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:53.748116 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:53.748160 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:53.751843 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:53.751885 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:53.751935 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:20:53.756255 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:53.757027 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:53.757815 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:53.765784 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.765818 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.765796 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.765988 18574 server_base.cc:1034] running on GCE node
I20260620 20:20:53.766388 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:53.766935 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:53.768106 18574 hybrid_clock.cc:648] HybridClock initialized: now 1781986853768083 us; error 40 us; skew 500 ppm
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:53.770597 18574 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:20:53.771330 18574 webserver.cc:460] Webserver started at http://127.2.235.194:44637/ using document root <none> and password file <none>
I20260620 20:20:53.771852 18574 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:53.771939 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:53.772142 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:53.773936 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "5a18d029108848359ab8e28d30f79432"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "78ef2bb0b697abcb9fef2e550f1b34dc"
server_key_iv: "b19da1bd97a31714ef26656b64d513a0"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.774430 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "5a18d029108848359ab8e28d30f79432"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "78ef2bb0b697abcb9fef2e550f1b34dc"
server_key_iv: "b19da1bd97a31714ef26656b64d513a0"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.777545 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:20:53.779637 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.780699 18574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:20:53.780817 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "5a18d029108848359ab8e28d30f79432"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "78ef2bb0b697abcb9fef2e550f1b34dc"
server_key_iv: "b19da1bd97a31714ef26656b64d513a0"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.780913 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:53.791635 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:53.794181 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:53.794346 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:53.794800 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:53.795451 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:53.795524 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.795604 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:53.795662 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.804232 18574 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:44265
I20260620 20:20:53.804261 18702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:44265 every 8 connection(s)
I20260620 20:20:53.805092 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:20:53.806264 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18574
I20260620 20:20:53.806366 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:20:53.806650 2991 external_mini_cluster.cc:1428] Setting key 52c5019a9cbd81e1b5c5047f25311ef6
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:20:53.818935 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.806700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:49819 (local address 127.2.235.254:42387)
0620 20:20:53.806841 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:53.806845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:53.807907 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 20:20:53.808744 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:53.808748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:53.808803 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:53.808881 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:53.810266 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.810762 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.812525 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.812745 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.812826 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.815374 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:53.815379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:53.815382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:53.815409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:53.817009 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.817516 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.817519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.817520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.817604 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:53.817989 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:53.817991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:53.817992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:53.818168 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:53.818242 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:53.818720 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 20:20:53.818820 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:20:53.819741 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:53.807059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:49819)
0620 20:20:53.807747 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:53.807795 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:53.808552 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:53.809043 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:53.809049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:53.809429 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:53.810115 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:53.810135 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.810897 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.810903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:53.812389 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:53.812405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:53.812891 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:53.812898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:53.813108 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:53.813660 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:53.813684 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:53.815208 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:53.817114 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:53.817118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:53.817127 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:53.817435 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:53.817714 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:53.817719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:53.817725 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:53.817900 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:53.818260 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:53.818264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:53.818477 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 20:20:53.818655 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 20:20:53.818923 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":127,"threads_started":1}
I20260620 20:20:53.820783 18703 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42387
I20260620 20:20:53.820986 18703 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:53.821456 18703 heartbeater.cc:511] Master 127.2.235.254:42387 requested a full tablet report, sending...
I20260620 20:20:53.822326 18378 ts_manager.cc:194] Registered new tserver with Master: 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
I20260620 20:20:53.822858 18378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:49819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:20:53.831707 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42387
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:20:53.931529 18708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:20:53.931720 18708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:20:53.931756 18708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:20:53.935163 18708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:20:53.935201 18708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:20:53.935249 18708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:20:53.940038 18708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:20:53.940798 18708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:20:53.941573 18708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:20:53.949663 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.949704 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:20:53.949680 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:20:53.949978 18708 server_base.cc:1034] running on GCE node
I20260620 20:20:53.950343 18708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:20:53.950863 18708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:20:53.952008 18708 hybrid_clock.cc:648] HybridClock initialized: now 1781986853951992 us; error 26 us; skew 500 ppm
Jun 20 20:20:53 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:20:53.954663 18708 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:20:53.955448 18708 webserver.cc:460] Webserver started at http://127.2.235.195:35339/ using document root <none> and password file <none>
I20260620 20:20:53.955981 18708 fs_manager.cc:359] Metadata directory not provided
I20260620 20:20:53.956036 18708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:20:53.956238 18708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:20:53.958039 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "906a6e3ad3af455db800b52af68e3961"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "3ef775d6d406cfef59bc44d7cf81d353"
server_key_iv: "bf380be8722c48ea6d83e29b4ff6227b"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.958583 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "906a6e3ad3af455db800b52af68e3961"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "3ef775d6d406cfef59bc44d7cf81d353"
server_key_iv: "bf380be8722c48ea6d83e29b4ff6227b"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.961826 18708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:20:53.964018 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:53.965139 18708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:20:53.965306 18708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "906a6e3ad3af455db800b52af68e3961"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "3ef775d6d406cfef59bc44d7cf81d353"
server_key_iv: "bf380be8722c48ea6d83e29b4ff6227b"
server_key_version: "encryptionkey@0"
I20260620 20:20:53.965422 18708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:20:54.004630 18708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:20:54.007542 18708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:20:54.007741 18708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:20:54.008311 18708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:20:54.009090 18708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:20:54.009159 18708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.009230 18708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:20:54.009300 18708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:20:54.017976 18708 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:41747
I20260620 20:20:54.018003 18836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:41747 every 8 connection(s)
I20260620 20:20:54.018826 18708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:20:54.026651 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18708
I20260620 20:20:54.026780 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:20:54.027070 2991 external_mini_cluster.cc:1428] Setting key 14dd5ffcfe2ce5c573966efde5abf979
Jun 20 20:20:54 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986853, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:20:54.031564 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.020555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:38103 (local address 127.2.235.254:42387)
0620 20:20:54.020696 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.020700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.021728 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 20:20:54.022551 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.022554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.022602 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.022674 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:54.023770 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.024279 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.025426 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.025641 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.025725 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.028337 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:54.028362 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:54.028368 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:54.028400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:54.029946 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:54.030363 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:54.030365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:54.030366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:54.030432 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:54.030720 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:54.030722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:54.030723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:54.030872 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:54.030941 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.031352 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.031454 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:20:54.032339 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.020867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:38103)
0620 20:20:54.021562 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.021620 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.022367 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.022851 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.022858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.023192 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:20:54.023648 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.023659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.024398 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.024400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.025316 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.025330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.025767 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.025770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.025934 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.026542 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:20:54.026558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:20:54.028166 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:20:54.030040 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:54.030044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:54.030053 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:54.030274 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:54.030523 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:20:54.030525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:20:54.030526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:20:54.030650 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:20:54.030948 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:20:54.030951 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:20:54.031158 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.031295 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.031527 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":108,"threads_started":1}
I20260620 20:20:54.033450 18837 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42387
I20260620 20:20:54.033661 18837 heartbeater.cc:464] Registering TS with master...
I20260620 20:20:54.034060 18837 heartbeater.cc:511] Master 127.2.235.254:42387 requested a full tablet report, sending...
I20260620 20:20:54.034971 18378 ts_manager.cc:194] Registered new tserver with Master: 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747)
I20260620 20:20:54.035471 18378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:38103
I20260620 20:20:54.040839 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:20:54.049103 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.042802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.2.235.254:42387)
0620 20:20:54.042908 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.042911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.043023 (+ 112us) server_negotiation.cc:407] Connection header received
0620 20:20:54.043098 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.043100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.043131 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.043191 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:20:54.043761 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.044204 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.044954 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.045144 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.045215 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.046448 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:20:54.046450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:20:54.046452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:20:54.046468 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:20:54.047677 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:54.048043 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:54.048045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:54.048046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:54.048110 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:20:54.048330 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:20:54.048332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:20:54.048333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:20:54.048508 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:20:54.048606 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.048919 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.049001 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:20:54.051800 18378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:20:54.054796 18378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:20:54.067943 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.062194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44265 (local address 127.0.0.1:57642)
0620 20:20:54.062579 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.062592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.062693 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.063965 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.063967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.063990 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:54.064129 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.064135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.065542 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.065548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.066568 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.066577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.067526 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.067528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.067600 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.067625 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.067702 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.067773 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":3,"spinlock_wait_cycles":20480,"thread_start_us":97,"threads_started":1}
I20260620 20:20:54.067960 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.063157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.2.235.194:44265)
0620 20:20:54.063550 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.063561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.063581 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:20:54.063654 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.063661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.063829 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.064270 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:54.064420 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.065429 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.066680 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.067420 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.067528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.067656 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.067728 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.067827 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
I20260620 20:20:54.068318 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.062791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:41747 (local address 127.0.0.1:58246)
0620 20:20:54.063005 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.063021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.063102 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.063959 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.063962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.063989 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:54.064150 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.064156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.065161 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.065164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.066854 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.066872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.067859 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.067862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.067945 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.067974 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.068094 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.068179 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":171,"spinlock_wait_cycles":1792,"thread_start_us":90,"threads_started":1}
I20260620 20:20:54.068944 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.062909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.2.235.195:41747)
0620 20:20:54.063480 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.063493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.063515 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:20:54.063580 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.063585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.063756 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.063888 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:54.064254 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.065034 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.066984 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.067731 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.067841 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.067950 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.068704 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.068788 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":123,"threads_started":1}
I20260620 20:20:54.069350 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.062125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38677 (local address 127.0.0.1:48314)
0620 20:20:54.062533 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.062578 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.062689 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.063769 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.063775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.063808 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:54.063929 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.063936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.065289 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.065292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.066878 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.066884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.068895 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.068899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.069003 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.069037 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.069112 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.069181 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":182,"thread_start_us":59,"threads_started":1}
I20260620 20:20:54.069391 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.062235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.2.235.193:38677)
0620 20:20:54.063249 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.063266 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.063290 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:20:54.063364 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.063369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.063555 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.063737 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:54.064052 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.065159 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.068014 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.068762 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.068881 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.069005 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.069150 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.069235 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":110,"threads_started":1}
I20260620 20:20:54.070901 18637 tablet_service.cc:1467] Processing CreateTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 (DEFAULT_TABLE table=test-table [id=6bc96c307cf8403586538de51f648b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:54.071772 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d00bd0e4a645a6be1079d612cd6c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:54.071702 18771 tablet_service.cc:1467] Processing CreateTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 (DEFAULT_TABLE table=test-table [id=6bc96c307cf8403586538de51f648b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:54.071761 18503 tablet_service.cc:1467] Processing CreateTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 (DEFAULT_TABLE table=test-table [id=6bc96c307cf8403586538de51f648b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:20:54.072659 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d00bd0e4a645a6be1079d612cd6c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:54.072669 18771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d00bd0e4a645a6be1079d612cd6c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:20:54.078086 18854 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Bootstrap starting.
I20260620 20:20:54.079247 18855 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Bootstrap starting.
I20260620 20:20:54.080241 18856 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap starting.
I20260620 20:20:54.080493 18854 tablet_bootstrap.cc:654] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:54.081339 18854 log.cc:824] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:54.081756 18855 tablet_bootstrap.cc:654] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:54.082382 18856 tablet_bootstrap.cc:654] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:20:54.082548 18855 log.cc:824] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:54.083433 18856 log.cc:824] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:20:54.083849 18854 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: No bootstrap required, opened a new log
I20260620 20:20:54.084035 18854 ts_tablet_manager.cc:1397] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:20:54.084735 18855 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: No bootstrap required, opened a new log
I20260620 20:20:54.084934 18855 ts_tablet_manager.cc:1397] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 20:20:54.085209 18856 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: No bootstrap required, opened a new log
I20260620 20:20:54.085443 18856 ts_tablet_manager.cc:1397] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260620 20:20:54.087728 18854 raft_consensus.cc:348] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.087981 18854 raft_consensus.cc:374] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:54.088027 18854 raft_consensus.cc:729] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a18d029108848359ab8e28d30f79432, State: Initialized, Role: FOLLOWER
I20260620 20:20:54.088466 18854 consensus_queue.cc:260] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.088522 18855 raft_consensus.cc:348] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.088760 18855 raft_consensus.cc:374] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:54.088835 18855 raft_consensus.cc:729] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906a6e3ad3af455db800b52af68e3961, State: Initialized, Role: FOLLOWER
I20260620 20:20:54.088829 18856 raft_consensus.cc:348] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.089141 18856 raft_consensus.cc:374] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:20:54.089206 18856 raft_consensus.cc:729] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daeb5f1388e741248570d0a0ceeab6a2, State: Initialized, Role: FOLLOWER
I20260620 20:20:54.089224 18855 consensus_queue.cc:260] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.089442 18854 ts_tablet_manager.cc:1428] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260620 20:20:54.089701 18856 consensus_queue.cc:260] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.089792 18703 heartbeater.cc:503] Master 127.2.235.254:42387 was elected leader, sending a full tablet report...
I20260620 20:20:54.090106 18855 ts_tablet_manager.cc:1428] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:20:54.090445 18837 heartbeater.cc:503] Master 127.2.235.254:42387 was elected leader, sending a full tablet report...
I20260620 20:20:54.090605 18856 ts_tablet_manager.cc:1428] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 20:20:54.090790 18569 heartbeater.cc:503] Master 127.2.235.254:42387 was elected leader, sending a full tablet report...
W20260620 20:20:54.120882 18570 tablet.cc:2369] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:20:54.187690 18861 raft_consensus.cc:482] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:20:54.187867 18861 raft_consensus.cc:504] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.188905 18861 leader_election.cc:288] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
I20260620 20:20:54.192534 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.189220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:48327 (local address 127.2.235.194:44265)
0620 20:20:54.189379 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.189383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.189395 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:20:54.189545 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.189548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.189600 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.189684 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:54.190038 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.190604 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.191596 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.192034 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.192114 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.192150 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.192368 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.192427 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:20:54.192538 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.189129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44265 (local address 127.2.235.195:48327)
0620 20:20:54.189289 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.189306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.189404 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.189704 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.189707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.189723 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:54.189905 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.189913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.190708 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.190711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.191490 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.191498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.192133 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.192136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.192232 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.192257 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.192337 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.192410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"spinlock_wait_cycles":4992}
I20260620 20:20:54.192745 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.189182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:34077 (local address 127.2.235.193:38677)
0620 20:20:54.189321 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.189326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.189771 (+ 445us) server_negotiation.cc:407] Connection header received
0620 20:20:54.189911 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.189914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.189974 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.190058 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:20:54.190339 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.190817 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.191831 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.192234 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.192348 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.192395 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.192568 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.192633 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:20:54.192755 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.189130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38677 (local address 127.2.235.195:34077)
0620 20:20:54.189672 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.189685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:20:54.189809 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:20:54.190076 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:20:54.190079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:20:54.190095 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:20:54.190196 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.190202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.190914 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.190916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.191702 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:20:54.191711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.192373 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.192376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:20:54.192457 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.192476 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:20:54.192543 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:20:54.192607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":158,"spinlock_wait_cycles":4736,"thread_start_us":145,"threads_started":1}
I20260620 20:20:54.193084 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d00bd0e4a645a6be1079d612cd6c69" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a18d029108848359ab8e28d30f79432" is_pre_election: true
I20260620 20:20:54.193359 18657 raft_consensus.cc:2454] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 0.
I20260620 20:20:54.193370 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d00bd0e4a645a6be1079d612cd6c69" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeb5f1388e741248570d0a0ceeab6a2" is_pre_election: true
I20260620 20:20:54.193583 18523 raft_consensus.cc:2454] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 0.
I20260620 20:20:54.193742 18725 leader_election.cc:302] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961; no voters:
I20260620 20:20:54.193979 18861 raft_consensus.cc:2781] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:20:54.194053 18861 raft_consensus.cc:482] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:20:54.194092 18861 raft_consensus.cc:3037] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:54.194934 18861 raft_consensus.cc:504] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.195186 18861 leader_election.cc:288] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 election: Requested vote from peers daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
I20260620 20:20:54.195457 18523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d00bd0e4a645a6be1079d612cd6c69" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeb5f1388e741248570d0a0ceeab6a2"
I20260620 20:20:54.195467 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d00bd0e4a645a6be1079d612cd6c69" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a18d029108848359ab8e28d30f79432"
I20260620 20:20:54.195549 18523 raft_consensus.cc:3037] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:54.195572 18657 raft_consensus.cc:3037] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:20:54.196452 18657 raft_consensus.cc:2454] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 1.
I20260620 20:20:54.196534 18523 raft_consensus.cc:2454] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 1.
I20260620 20:20:54.196750 18725 leader_election.cc:302] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961; no voters:
I20260620 20:20:54.196933 18861 raft_consensus.cc:2781] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:20:54.197116 18861 raft_consensus.cc:686] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 1 LEADER]: Becoming Leader. State: Replica: 906a6e3ad3af455db800b52af68e3961, State: Running, Role: LEADER
I20260620 20:20:54.197448 18861 consensus_queue.cc:237] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:20:54.200191 18377 catalog_manager.cc:5495] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 906a6e3ad3af455db800b52af68e3961 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "906a6e3ad3af455db800b52af68e3961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } health_report { overall_health: HEALTHY } } }
I20260620 20:20:54.220240 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.216383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.2.235.195:41747)
0620 20:20:54.216512 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:20:54.216517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:20:54.216576 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:20:54.216776 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:20:54.216779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:20:54.216849 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:20:54.216947 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:20:54.217224 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.217725 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.218616 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:20:54.218867 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:20:54.218981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:20:54.219296 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:20:54.219831 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:20:54.219957 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:20:54.220014 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 20:20:54.220079 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:20:54.225337 18523 raft_consensus.cc:1261] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Refusing update from remote peer 906a6e3ad3af455db800b52af68e3961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:54.225358 18657 raft_consensus.cc:1261] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Refusing update from remote peer 906a6e3ad3af455db800b52af68e3961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:20:54.225960 18864 consensus_queue.cc:1035] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:54.226150 18861 consensus_queue.cc:1035] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:20:54.233784 18869 mvcc.cc:205] Tried to move back new op lower bound from 7299018154900897792 to 7299018154794958848. Current Snapshot: MvccSnapshot[applied={T|T < 7299018154900897792}]
I20260620 20:20:54.241449 18377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:54.241644 18377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:20:54.243881 18377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Sending DeleteTablet for 3 replicas of tablet 97d00bd0e4a645a6be1079d612cd6c69
I20260620 20:20:54.244570 18503 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:54 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:48314
I20260620 20:20:54.244659 18637 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:54 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:57642
I20260620 20:20:54.244720 18771 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:54 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:58246
I20260620 20:20:54.245187 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18440
I20260620 20:20:54.245209 18879 tablet_replica.cc:330] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: stopping tablet replica
I20260620 20:20:54.246053 18879 raft_consensus.cc:2229] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:20:54.246460 18879 raft_consensus.cc:2258] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:54.247951 18879 ts_tablet_manager.cc:1905] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:20:54.251448 18879 ts_tablet_manager.cc:1918] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:20:54.251561 18879 log.cc:1196] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/97d00bd0e4a645a6be1079d612cd6c69
W20260620 20:20:54.251914 18365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:38677 (error 108)
I20260620 20:20:54.251963 18879 ts_tablet_manager.cc:1939] T 97d00bd0e4a645a6be1079d612cd6c69 P 906a6e3ad3af455db800b52af68e3961: Deleting consensus metadata
I20260620 20:20:54.252279 18878 tablet_replica.cc:330] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: stopping tablet replica
I20260620 20:20:54.252522 18878 raft_consensus.cc:2229] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:20:54.252754 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18574
I20260620 20:20:54.252812 18878 raft_consensus.cc:2258] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:20:54.254303 18363 catalog_manager.cc:4841] TS 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): tablet 97d00bd0e4a645a6be1079d612cd6c69 (table test-table [id=6bc96c307cf8403586538de51f648b4f]) successfully deleted
I20260620 20:20:54.254665 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.253619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38677 (local address 127.0.0.1:48318)
0620 20:20:54.254548 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.254604 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:38677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":859}
W20260620 20:20:54.255244 18365 catalog_manager.cc:4584] TS daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97d00bd0e4a645a6be1079d612cd6c69: Network error: Client connection negotiation failed: client connection to 127.2.235.193:38677: connect: Connection refused (error 111)
I20260620 20:20:54.259474 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18708
I20260620 20:20:54.260308 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:20:54.259723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44265 (local address 127.0.0.1:57646)
0620 20:20:54.260158 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 20:20:54.260217 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.194:44265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355}
I20260620 20:20:54.264473 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18346
I20260620 20:21:14.270635 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42387
--webserver_interface=127.2.235.254
--webserver_port=43017
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:14.375043 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:14.375265 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:14.375303 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:14.378878 18881 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:21:14.378924 18881 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:21:14.378942 18881 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:21:14.378957 18881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:14.378975 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:14.378993 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:14.379010 18881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:14.379026 18881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:14.383798 18881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:14.384596 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:14.385516 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:14.397636 18890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.397588 18888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:14.397676 18881 server_base.cc:1034] running on GCE node
W20260620 20:21:14.397588 18887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:14.398404 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:14.399379 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:14.400565 18881 hybrid_clock.cc:648] HybridClock initialized: now 1781986874400548 us; error 30 us; skew 500 ppm
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:14.407490 18881 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:21:14.408324 18881 webserver.cc:460] Webserver started at http://127.2.235.254:43017/ using document root <none> and password file <none>
I20260620 20:21:14.408792 18881 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:14.408854 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:14.412305 18881 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:14.414311 18897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:14.415436 18881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:14.415587 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "1e567951b53b4219928daf572f461847"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "985def8a1e700a752bf126e529a69bfc"
server_key_iv: "d113450d65c1371476b5b3233c8401b7"
server_key_version: "encryptionkey@0"
I20260620 20:21:14.416122 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:14.429026 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:14.432044 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:14.432183 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:14.438774 18881 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:42387
I20260620 20:21:14.439046 18949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:42387 every 8 connection(s)
I20260620 20:21:14.439779 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:14.443915 18950 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:21:14.445355 18950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap starting.
I20260620 20:21:14.446458 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18881
I20260620 20:21:14.446870 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:38677
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:42387
--webserver_port=45559
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:14.461401 18950 log.cc:824] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:14.468809 18950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:21:14.469133 18950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap complete.
I20260620 20:21:14.473034 18950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:14.473416 18950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e567951b53b4219928daf572f461847, State: Initialized, Role: FOLLOWER
I20260620 20:21:14.473834 18950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:14.473930 18950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:14.474007 18950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:14.474107 18950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:21:14.475884 18950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:14.476176 18950 leader_election.cc:302] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e567951b53b4219928daf572f461847; no voters:
I20260620 20:21:14.476449 18950 leader_election.cc:288] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:21:14.476531 18954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:21:14.476750 18954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 LEADER]: Becoming Leader. State: Replica: 1e567951b53b4219928daf572f461847, State: Running, Role: LEADER
I20260620 20:21:14.477077 18954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:14.477622 18950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:14.477944 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e567951b53b4219928daf572f461847. Latest consensus state: current_term: 2 leader_uuid: "1e567951b53b4219928daf572f461847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } } }
I20260620 20:21:14.478113 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:14.478302 18955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1e567951b53b4219928daf572f461847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } } }
I20260620 20:21:14.478401 18955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:14.478686 18958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:14.483127 18958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:14.484601 18958 catalog_manager.cc:1259] Loaded cluster ID: 833b3666744645a38fe6430578d96809
I20260620 20:21:14.484689 18958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:14.487279 18958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:14.488006 18958 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Loaded TSK: 0
I20260620 20:21:14.501639 18958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Generated new TSK 1
I20260620 20:21:14.501855 18958 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:21:14.576061 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:14.576330 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:14.576388 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:14.581866 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:14.581924 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:14.582003 18952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:14.588380 18952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:38677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=45559
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:14.589332 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:14.590377 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:14.599567 18978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.599567 18977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.599673 18980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:14.599953 18952 server_base.cc:1034] running on GCE node
I20260620 20:21:14.600345 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:14.601032 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:14.602219 18952 hybrid_clock.cc:648] HybridClock initialized: now 1781986874602171 us; error 57 us; skew 500 ppm
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:14.605188 18952 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:21:14.606032 18952 webserver.cc:460] Webserver started at http://127.2.235.193:45559/ using document root <none> and password file <none>
I20260620 20:21:14.606541 18952 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:14.606607 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:14.610177 18952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:14.612197 18987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:14.613479 18952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:14.613577 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "daeb5f1388e741248570d0a0ceeab6a2"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "38febb8697d45d2f0743d8e493d3a75c"
server_key_iv: "5de8419c6f691847ea178f40dde3537a"
server_key_version: "encryptionkey@0"
I20260620 20:21:14.614045 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:14.635221 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:14.638211 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:14.638407 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:14.638976 18952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:14.639964 18994 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:21:14.643864 18952 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:21:14.643918 18952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 20:21:14.643976 18952 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:21:14.645494 18952 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:21:14.645545 18952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:14.645783 18994 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap starting.
I20260620 20:21:14.655710 18952 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:38677
I20260620 20:21:14.655730 19101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:38677 every 8 connection(s)
I20260620 20:21:14.656777 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:14.665055 18994 log.cc:824] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:14.666922 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 18952
I20260620 20:21:14.667364 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:44265
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:42387
--webserver_port=44637
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:14.673395 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.658579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50525 (local address 127.2.235.254:42387)
0620 20:21:14.659357 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:14.659369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:14.659693 (+ 324us) server_negotiation.cc:407] Connection header received
0620 20:21:14.660636 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:14.660652 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:14.661032 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:14.661441 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:14.662326 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.663789 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.664813 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.665521 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.665641 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.668415 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:14.668424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:14.668441 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:14.668475 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:14.670231 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:14.670854 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:14.670859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:14.670863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:14.670975 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:14.671334 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:14.671337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:14.671339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:14.671724 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:14.671844 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:14.672317 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 20:21:14.672569 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":91,"threads_started":1}
I20260620 20:21:14.673413 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.658562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:50525)
0620 20:21:14.659552 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:14.659585 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:14.660206 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:14.661452 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:14.661458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:14.661767 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:14.662163 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.662175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.663929 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.663931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.664696 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.664704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.665659 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.665663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.665836 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.666468 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:14.666483 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:14.668213 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:14.670365 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:14.670371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:14.670383 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:14.670743 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:14.671087 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:14.671090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:14.671092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:14.671228 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:14.671834 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:14.671839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:14.672042 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 20:21:14.672217 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 20:21:14.672534 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":107,"threads_started":1}
I20260620 20:21:14.673612 18994 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:21:14.674115 18994 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap complete.
I20260620 20:21:14.675529 18994 ts_tablet_manager.cc:1397] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.011s
I20260620 20:21:14.675683 19102 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42387
I20260620 20:21:14.675886 19102 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:14.676302 19102 heartbeater.cc:511] Master 127.2.235.254:42387 requested a full tablet report, sending...
I20260620 20:21:14.678578 18914 ts_manager.cc:194] Registered new tserver with Master: daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677)
I20260620 20:21:14.679039 18994 raft_consensus.cc:348] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:21:14.679613 18994 raft_consensus.cc:729] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daeb5f1388e741248570d0a0ceeab6a2, State: Initialized, Role: FOLLOWER
I20260620 20:21:14.680133 18994 consensus_queue.cc:260] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:21:14.681072 18994 ts_tablet_manager.cc:1428] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260620 20:21:14.682034 18914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:50525
I20260620 20:21:14.684329 19102 heartbeater.cc:503] Master 127.2.235.254:42387 was elected leader, sending a full tablet report...
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.193@KRBTEST.COM
I20260620 20:21:14.691603 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.681493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38677 (local address 127.0.0.1:39814)
0620 20:21:14.682050 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:14.682095 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:14.682265 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:14.682739 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:14.682745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:14.683104 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:14.683209 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.683216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.684537 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.684540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.685507 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.685518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.686292 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.686295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.686389 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.686871 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:14.686891 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:14.688274 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:14.690112 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:14.690120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:14.690124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:14.690350 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:14.690640 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:14.690642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:14.690644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:14.690713 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:14.691158 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:14.691162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:14.691247 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 20:21:14.691347 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:21:14.691488 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":105,"threads_started":1}
I20260620 20:21:14.691790 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.681476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.2.235.193:38677)
0620 20:21:14.681793 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:14.681802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:14.682173 (+ 371us) server_negotiation.cc:407] Connection header received
0620 20:21:14.682429 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:14.682435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:14.682601 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:14.682723 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:14.683336 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.684421 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.685631 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.686099 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.686164 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.688389 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:14.688398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:14.688409 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:14.688431 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:14.690014 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:14.690460 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:14.690465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:14.690467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:14.690553 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:14.690793 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:14.690796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:14.690798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:14.691055 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:14.691176 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:14.691446 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 20:21:14.691611 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20260620 20:21:14.692592 19036 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:54 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:39814
I20260620 20:21:14.693403 19111 tablet_replica.cc:330] stopping tablet replica
I20260620 20:21:14.693643 19111 raft_consensus.cc:2229] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:14.693857 19111 raft_consensus.cc:2258] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:14.694630 19111 ts_tablet_manager.cc:1905] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:14.697724 19111 ts_tablet_manager.cc:1918] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:14.697822 19111 log.cc:1196] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal/wals/97d00bd0e4a645a6be1079d612cd6c69
I20260620 20:21:14.698084 19111 ts_tablet_manager.cc:1939] T 97d00bd0e4a645a6be1079d612cd6c69 P daeb5f1388e741248570d0a0ceeab6a2: Deleting consensus metadata
I20260620 20:21:14.699450 18901 catalog_manager.cc:4841] TS daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677): tablet 97d00bd0e4a645a6be1079d612cd6c69 (table test-table [id=6bc96c307cf8403586538de51f648b4f]) successfully deleted
W20260620 20:21:14.775357 19107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:14.775553 19107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:14.775588 19107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:14.779141 19107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:14.779184 19107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:14.779243 19107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:14.783921 19107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:44265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=44637
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:14.784646 19107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:14.785452 19107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:14.793604 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.793604 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.793622 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:14.793665 19107 server_base.cc:1034] running on GCE node
I20260620 20:21:14.794171 19107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:14.794740 19107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:14.795913 19107 hybrid_clock.cc:648] HybridClock initialized: now 1781986874795907 us; error 27 us; skew 500 ppm
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:14.798288 19107 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:21:14.799014 19107 webserver.cc:460] Webserver started at http://127.2.235.194:44637/ using document root <none> and password file <none>
I20260620 20:21:14.799512 19107 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:14.799607 19107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:14.802984 19107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:14.804742 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:14.805802 19107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:14.805913 19107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "5a18d029108848359ab8e28d30f79432"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "78ef2bb0b697abcb9fef2e550f1b34dc"
server_key_iv: "b19da1bd97a31714ef26656b64d513a0"
server_key_version: "encryptionkey@0"
I20260620 20:21:14.806373 19107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:14.832178 19107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:14.834689 19107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:14.835053 19107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:14.835597 19107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:14.836563 19133 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:21:14.840142 19107 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:21:14.840214 19107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 20:21:14.840301 19107 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:21:14.841898 19107 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:21:14.841964 19107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:14.842180 19133 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Bootstrap starting.
I20260620 20:21:14.851694 19107 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:44265
I20260620 20:21:14.851750 19240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:44265 every 8 connection(s)
I20260620 20:21:14.852480 19107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:14.854728 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19107
I20260620 20:21:14.855244 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:41747
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:42387
--webserver_port=35339
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:14.860144 19133 log.cc:824] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:14.866230 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.854191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:55985 (local address 127.2.235.254:42387)
0620 20:21:14.854334 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:14.854338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:14.854796 (+ 458us) server_negotiation.cc:407] Connection header received
0620 20:21:14.855810 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:14.855813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:14.855872 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:14.855970 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:14.857119 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.857989 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.859157 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.859470 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.859576 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.862215 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:14.862221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:14.862225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:14.862258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:14.863992 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:14.864529 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:14.864534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:14.864536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:14.864630 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:14.864976 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:14.864980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:14.864984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:14.865216 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:14.865378 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:14.865876 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 20:21:14.866017 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:21:14.866828 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.854272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:55985)
0620 20:21:14.854670 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:14.854713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:14.855615 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:14.856113 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:14.856120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:14.856439 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:14.856960 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.856980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.858156 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.858159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.859039 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.859047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.859578 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.859581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.859714 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.860272 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:14.860295 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:14.862049 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:14.864108 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:14.864113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:14.864123 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:14.864428 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:14.864727 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:14.864730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:14.864731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:14.864877 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:14.865388 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:14.865393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:14.865636 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 20:21:14.865806 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 20:21:14.866064 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":128,"threads_started":1}
I20260620 20:21:14.867321 19133 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:21:14.867743 19133 tablet_bootstrap.cc:492] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Bootstrap complete.
I20260620 20:21:14.867901 19241 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42387
I20260620 20:21:14.868129 19241 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:14.868649 19241 heartbeater.cc:511] Master 127.2.235.254:42387 requested a full tablet report, sending...
I20260620 20:21:14.868757 19133 ts_tablet_manager.cc:1397] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20260620 20:21:14.870522 18914 ts_manager.cc:194] Registered new tserver with Master: 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
I20260620 20:21:14.870863 19133 raft_consensus.cc:348] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:21:14.871253 19133 raft_consensus.cc:729] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a18d029108848359ab8e28d30f79432, State: Initialized, Role: FOLLOWER
I20260620 20:21:14.871708 18914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:55985
I20260620 20:21:14.872018 19133 consensus_queue.cc:260] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } }
I20260620 20:21:14.872871 19133 ts_tablet_manager.cc:1428] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260620 20:21:14.873759 19241 heartbeater.cc:503] Master 127.2.235.254:42387 was elected leader, sending a full tablet report...
I20260620 20:21:14.876473 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.871418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44265 (local address 127.0.0.1:59622)
0620 20:21:14.871718 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:14.871733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:14.871805 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:14.873536 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:14.873539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:14.873555 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:14.873647 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.873657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.874488 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.874494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.875371 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:14.875386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.876055 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.876058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:14.876154 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.876202 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 20:21:14.876292 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:21:14.876359 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260620 20:21:14.876482 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:14.871676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.2.235.194:44265)
0620 20:21:14.873164 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:14.873171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:14.873186 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:21:14.873230 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:14.873233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:14.873407 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:14.873545 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:14.873744 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.874369 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.875493 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:14.875945 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:14.876054 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:14.876164 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:14.876317 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:21:14.876387 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":85,"threads_started":1}
I20260620 20:21:14.877357 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d00bd0e4a645a6be1079d612cd6c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:20:54 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:59622
I20260620 20:21:14.878218 19248 tablet_replica.cc:330] stopping tablet replica
I20260620 20:21:14.878513 19248 raft_consensus.cc:2229] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:14.878760 19248 raft_consensus.cc:2258] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:14.879628 19248 ts_tablet_manager.cc:1905] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:14.882776 19248 ts_tablet_manager.cc:1918] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:14.882876 19248 log.cc:1196] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/97d00bd0e4a645a6be1079d612cd6c69
I20260620 20:21:14.883149 19248 ts_tablet_manager.cc:1939] T 97d00bd0e4a645a6be1079d612cd6c69 P 5a18d029108848359ab8e28d30f79432: Deleting consensus metadata
I20260620 20:21:14.884387 18899 catalog_manager.cc:4841] TS 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): tablet 97d00bd0e4a645a6be1079d612cd6c69 (table test-table [id=6bc96c307cf8403586538de51f648b4f]) successfully deleted
W20260620 20:21:14.965485 19244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:14.965708 19244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:14.965746 19244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:14.969300 19244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:14.969357 19244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:14.969480 19244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:14.974304 19244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=35339
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:14.975077 19244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:14.975888 19244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:14.984586 19253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.984623 19254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:14.984731 19256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:14.984618 19244 server_base.cc:1034] running on GCE node
I20260620 20:21:14.985175 19244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:14.985750 19244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:14.986927 19244 hybrid_clock.cc:648] HybridClock initialized: now 1781986874986916 us; error 22 us; skew 500 ppm
Jun 20 20:21:14 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:14.989209 19244 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:21:14.989929 19244 webserver.cc:460] Webserver started at http://127.2.235.195:35339/ using document root <none> and password file <none>
I20260620 20:21:14.990476 19244 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:14.990546 19244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:14.993779 19244 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:14.995684 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:14.996860 19244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:14.996958 19244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "906a6e3ad3af455db800b52af68e3961"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "3ef775d6d406cfef59bc44d7cf81d353"
server_key_iv: "bf380be8722c48ea6d83e29b4ff6227b"
server_key_version: "encryptionkey@0"
I20260620 20:21:14.997440 19244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:15.009698 19244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:15.012151 19244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:15.012313 19244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:15.012776 19244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:15.013497 19244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:15.013557 19244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:15.013615 19244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:15.013661 19244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:15.022477 19244 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:41747
I20260620 20:21:15.022498 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:41747 every 8 connection(s)
I20260620 20:21:15.023296 19244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:21:15.024071 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19244
Jun 20 20:21:15 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986874, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:15.033011 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.025057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.2.235.254:42387)
0620 20:21:15.025347 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.025352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.025367 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:21:15.025421 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.025423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.025460 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.025561 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:15.026327 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.027118 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.028026 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.028316 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.028411 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.029791 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:15.029795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:15.029797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:15.029819 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:15.031384 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:15.031856 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:15.031859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:15.031860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:15.031927 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:15.032229 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:15.032232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:15.032234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:15.032435 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:15.032642 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.032736 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.032859 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260620 20:21:15.036530 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.025141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:48229 (local address 127.2.235.254:42387)
0620 20:21:15.025347 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.025352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.026224 (+ 872us) server_negotiation.cc:407] Connection header received
0620 20:21:15.027226 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.027229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.027280 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.027364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:15.028430 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.029086 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.030439 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.030652 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.030760 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.033456 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:15.033460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:15.033466 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:15.033494 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:15.034961 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:15.035405 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:15.035407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:15.035408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:15.035471 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:15.035759 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:15.035762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:15.035763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:15.035907 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:15.035973 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.036348 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.036430 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260620 20:21:15.037217 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.025106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:48229)
0620 20:21:15.026077 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:15.026108 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:15.027007 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:15.027515 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:15.027520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:15.027800 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:15.028284 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.028299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.029220 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.029223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.030293 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.030305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.030797 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.030801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.030983 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.031493 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:15.031512 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:15.033248 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:15.035052 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:15.035056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:15.035065 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:15.035288 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:15.035566 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:15.035568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:15.035570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:15.035680 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:15.035984 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:15.035988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:15.036164 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 20:21:15.036290 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 20:21:15.036486 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":126,"threads_started":1}
I20260620 20:21:15.038153 19377 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42387
I20260620 20:21:15.038340 19377 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:15.038690 19377 heartbeater.cc:511] Master 127.2.235.254:42387 requested a full tablet report, sending...
I20260620 20:21:15.039494 18914 ts_manager.cc:194] Registered new tserver with Master: 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747)
I20260620 20:21:15.039955 18914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:48229
I20260620 20:21:15.046202 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:21:15.049938 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.046808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.2.235.254:42387)
0620 20:21:15.046957 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.046961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.047090 (+ 129us) server_negotiation.cc:407] Connection header received
0620 20:21:15.047251 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.047254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.047338 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.047440 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:15.047661 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.048154 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.048910 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.049117 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.049210 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.049518 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:15.049614 (+ 96us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 20:21:15.049830 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57142: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 20:21:15.050011 19380 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57142: token expired
W20260620 20:21:15.050248 18841 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.235.254:42387)
I20260620 20:21:15.056774 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.050638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.2.235.254:42387)
0620 20:21:15.050739 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.050746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.050791 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:21:15.050905 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.050908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.050942 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.051029 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:15.051595 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.052022 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.052713 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.052917 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.053012 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.054115 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:15.054119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:15.054121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:15.054142 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:15.055266 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:15.055685 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:15.055688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:15.055690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:15.055766 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:15.056027 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:15.056030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:15.056032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:15.056184 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:15.056312 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.056533 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.056638 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:21:15.058732 18914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:21:15.061815 18914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:21:15.068451 19175 tablet_service.cc:1467] Processing CreateTablet for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e (DEFAULT_TABLE table=test-table [id=6e0b2eeeee534078b8019b804119fc46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:15.068660 19036 tablet_service.cc:1467] Processing CreateTablet for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e (DEFAULT_TABLE table=test-table [id=6e0b2eeeee534078b8019b804119fc46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:15.069002 19175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:15.069190 19036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:15.071480 19133 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Bootstrap starting.
I20260620 20:21:15.071846 18994 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap starting.
I20260620 20:21:15.072220 19133 tablet_bootstrap.cc:654] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:15.072587 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.067705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:41747 (local address 127.0.0.1:53450)
0620 20:21:15.067809 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:15.067822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:15.067893 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:15.068642 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:15.068645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:15.068663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:15.068732 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.068738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.070091 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.070094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.071148 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.071158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.072204 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.072209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.072308 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.072330 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:21:15.072404 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:21:15.072474 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 20:21:15.072631 18994 tablet_bootstrap.cc:654] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:15.073045 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.067848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.2.235.195:41747)
0620 20:21:15.068124 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.068137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.068157 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:21:15.068216 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.068220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.068462 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.068713 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:15.068930 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.069943 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.071297 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.072079 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.072181 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.072302 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.072829 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.072916 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
I20260620 20:21:15.073685 19133 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: No bootstrap required, opened a new log
I20260620 20:21:15.073776 19133 ts_tablet_manager.cc:1397] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:15.074333 19133 raft_consensus.cc:348] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.074429 19133 raft_consensus.cc:374] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:15.074453 19133 raft_consensus.cc:729] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a18d029108848359ab8e28d30f79432, State: Initialized, Role: FOLLOWER
I20260620 20:21:15.074520 18994 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: No bootstrap required, opened a new log
I20260620 20:21:15.074590 18994 ts_tablet_manager.cc:1397] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 20:21:15.074574 19133 consensus_queue.cc:260] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.074826 19133 ts_tablet_manager.cc:1428] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:21:15.074960 18994 raft_consensus.cc:348] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.075049 18994 raft_consensus.cc:374] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:15.075080 18994 raft_consensus.cc:729] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daeb5f1388e741248570d0a0ceeab6a2, State: Initialized, Role: FOLLOWER
I20260620 20:21:15.075193 18994 consensus_queue.cc:260] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.075418 18994 ts_tablet_manager.cc:1428] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:21:15.075671 19311 tablet_service.cc:1467] Processing CreateTablet for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e (DEFAULT_TABLE table=test-table [id=6e0b2eeeee534078b8019b804119fc46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:15.076668 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:15.083499 19389 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Bootstrap starting.
I20260620 20:21:15.085503 19389 tablet_bootstrap.cc:654] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:15.086294 19389 log.cc:824] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:15.087802 19389 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: No bootstrap required, opened a new log
I20260620 20:21:15.087980 19389 ts_tablet_manager.cc:1397] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 20:21:15.090629 19389 raft_consensus.cc:348] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.090821 19389 raft_consensus.cc:374] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:15.090907 19389 raft_consensus.cc:729] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906a6e3ad3af455db800b52af68e3961, State: Initialized, Role: FOLLOWER
I20260620 20:21:15.091296 19389 consensus_queue.cc:260] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.091869 19377 heartbeater.cc:503] Master 127.2.235.254:42387 was elected leader, sending a full tablet report...
I20260620 20:21:15.092233 19389 ts_tablet_manager.cc:1428] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20260620 20:21:15.103196 19242 tablet.cc:2369] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:21:15.158109 19103 tablet.cc:2369] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:15.158768 19391 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:15.158933 19391 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.159819 19391 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265), daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677)
I20260620 20:21:15.163326 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.160006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38677 (local address 127.2.235.195:60375)
0620 20:21:15.160123 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:15.160139 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:15.160240 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:15.160559 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:15.160562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:15.160580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:15.160710 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.160716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.161540 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.161557 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.162256 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.162263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.162871 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.162874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.162983 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.163004 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:21:15.163096 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 20:21:15.163160 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1,"spinlock_wait_cycles":4352}
I20260620 20:21:15.163403 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.160086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:60375 (local address 127.2.235.193:38677)
0620 20:21:15.160201 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.160206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.160221 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:21:15.160351 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.160354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.160428 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.160549 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:15.160828 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.161433 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.162366 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.162743 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.162879 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.163020 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.163188 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.163269 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:21:15.163518 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.160156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:36571 (local address 127.2.235.194:44265)
0620 20:21:15.160288 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.160294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.160594 (+ 300us) server_negotiation.cc:407] Connection header received
0620 20:21:15.160748 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.160751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.160798 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.160879 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:15.161133 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.161665 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.162559 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.162933 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.163079 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.163125 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.163332 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.163414 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:21:15.163518 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.160006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44265 (local address 127.2.235.195:36571)
0620 20:21:15.160505 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:15.160515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:15.160641 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:15.160896 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:15.160899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:15.160914 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:15.161028 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.161033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.161761 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.161765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.162450 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:15.162457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.163102 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.163105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:15.163185 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.163221 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:21:15.163302 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 20:21:15.163369 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":193,"spinlock_wait_cycles":26112,"thread_start_us":114,"threads_started":1}
I20260620 20:21:15.163924 19056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c1ab79c6f384b7589c3ef3f7fb77e1e" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeb5f1388e741248570d0a0ceeab6a2" is_pre_election: true
I20260620 20:21:15.163990 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c1ab79c6f384b7589c3ef3f7fb77e1e" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a18d029108848359ab8e28d30f79432" is_pre_election: true
I20260620 20:21:15.164167 19056 raft_consensus.cc:2454] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 0.
I20260620 20:21:15.164240 19195 raft_consensus.cc:2454] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 0.
I20260620 20:21:15.164618 19265 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961; no voters:
I20260620 20:21:15.164825 19391 raft_consensus.cc:2781] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:15.164904 19391 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:15.164944 19391 raft_consensus.cc:3037] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:15.165900 19391 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.166213 19391 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 election: Requested vote from peers 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265), daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677)
I20260620 20:21:15.166530 19195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c1ab79c6f384b7589c3ef3f7fb77e1e" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a18d029108848359ab8e28d30f79432"
I20260620 20:21:15.166642 19195 raft_consensus.cc:3037] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:15.166664 19056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c1ab79c6f384b7589c3ef3f7fb77e1e" candidate_uuid: "906a6e3ad3af455db800b52af68e3961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeb5f1388e741248570d0a0ceeab6a2"
I20260620 20:21:15.166810 19056 raft_consensus.cc:3037] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:15.167635 19195 raft_consensus.cc:2454] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 1.
I20260620 20:21:15.167908 19265 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961; no voters:
I20260620 20:21:15.167958 19056 raft_consensus.cc:2454] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 906a6e3ad3af455db800b52af68e3961 in term 1.
I20260620 20:21:15.168067 19391 raft_consensus.cc:2781] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:15.168277 19391 raft_consensus.cc:686] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 1 LEADER]: Becoming Leader. State: Replica: 906a6e3ad3af455db800b52af68e3961, State: Running, Role: LEADER
I20260620 20:21:15.168541 19391 consensus_queue.cc:237] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:15.171253 18913 catalog_manager.cc:5495] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 reported cstate change: term changed from 0 to 1, leader changed from <none> to 906a6e3ad3af455db800b52af68e3961 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "906a6e3ad3af455db800b52af68e3961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } health_report { overall_health: UNKNOWN } } }
I20260620 20:21:15.194633 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:15.191332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.2.235.195:41747)
0620 20:21:15.191445 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:15.191449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:15.191511 (+ 62us) server_negotiation.cc:407] Connection header received
0620 20:21:15.191696 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:15.191698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:15.191741 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:15.191836 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:15.192066 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.192516 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.193311 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:15.193514 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:15.193618 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:15.193898 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:15.194254 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:15.194376 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:15.194446 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 20:21:15.194504 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:21:15.199534 19056 raft_consensus.cc:1261] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Refusing update from remote peer 906a6e3ad3af455db800b52af68e3961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:15.199539 19195 raft_consensus.cc:1261] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Refusing update from remote peer 906a6e3ad3af455db800b52af68e3961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:15.200116 19393 consensus_queue.cc:1035] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:15.200290 19391 consensus_queue.cc:1035] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:15.204895 19395 mvcc.cc:205] Tried to move back new op lower bound from 7299018240811593728 to 7299018240692502528. Current Snapshot: MvccSnapshot[applied={T|T < 7299018240811593728}]
I20260620 20:21:15.209614 19397 mvcc.cc:205] Tried to move back new op lower bound from 7299018240811593728 to 7299018240692502528. Current Snapshot: MvccSnapshot[applied={T|T < 7299018240811593728 or (T in {7299018240811593728})}]
I20260620 20:21:15.213068 18913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:15.213207 18913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:15.215286 18913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Sending DeleteTablet for 3 replicas of tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:15.215888 19175 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:15 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:59622
I20260620 20:21:15.215886 19311 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:15 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:53450
I20260620 20:21:15.216017 19036 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:15 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:39814
I20260620 20:21:15.216046 19248 tablet_replica.cc:330] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: stopping tablet replica
I20260620 20:21:15.216146 19248 raft_consensus.cc:2229] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:15.216305 19248 raft_consensus.cc:2258] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:15.216405 19404 tablet_replica.cc:330] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: stopping tablet replica
I20260620 20:21:15.216423 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18952
I20260620 20:21:15.216660 19404 raft_consensus.cc:2229] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:21:15.216944 19404 raft_consensus.cc:2258] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:15.217342 19248 ts_tablet_manager.cc:1905] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:15.219625 19248 ts_tablet_manager.cc:1918] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:15.219676 19248 log.cc:1196] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:15.219856 19248 ts_tablet_manager.cc:1939] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 5a18d029108848359ab8e28d30f79432: Deleting consensus metadata
I20260620 20:21:15.220618 18899 catalog_manager.cc:4841] TS 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e (table test-table [id=6e0b2eeeee534078b8019b804119fc46]) successfully deleted
I20260620 20:21:15.221138 19404 ts_tablet_manager.cc:1905] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:15.223587 19404 ts_tablet_manager.cc:1918] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:15.223675 19404 log.cc:1196] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:15.224005 19404 ts_tablet_manager.cc:1939] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P 906a6e3ad3af455db800b52af68e3961: Deleting consensus metadata
W20260620 20:21:15.224556 18901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:38677 (error 108)
I20260620 20:21:15.224969 18899 catalog_manager.cc:4841] TS 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e (table test-table [id=6e0b2eeeee534078b8019b804119fc46]) successfully deleted
W20260620 20:21:15.225593 19110 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:15.225313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38677 (local address 127.0.0.1:39820)
0620 20:21:15.225436 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:15.225513 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:38677: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:21:15.225660 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 19107
W20260620 20:21:15.225740 18901 catalog_manager.cc:4584] TS daeb5f1388e741248570d0a0ceeab6a2 (127.2.235.193:38677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c1ab79c6f384b7589c3ef3f7fb77e1e: Network error: Client connection negotiation failed: client connection to 127.2.235.193:38677: connect: Connection reset by peer (error 104)
I20260620 20:21:15.231910 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 19244
I20260620 20:21:15.236660 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 18881
I20260620 20:21:35.243178 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42387
--webserver_interface=127.2.235.254
--webserver_port=43017
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:35.342813 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:35.343045 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:35.343109 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:35.346782 19407 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:21:35.346839 19407 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:21:35.346864 19407 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:21:35.346890 19407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:35.346925 19407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:35.346958 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:35.347004 19407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:35.347039 19407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:35.351548 19407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:35.352391 19407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:35.353291 19407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:35.361212 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.361224 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.361287 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.361299 19407 server_base.cc:1034] running on GCE node
I20260620 20:21:35.361948 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:35.362905 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:35.364100 19407 hybrid_clock.cc:648] HybridClock initialized: now 1781986895364071 us; error 47 us; skew 500 ppm
Jun 20 20:21:35 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:35.366796 19407 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:21:35.367574 19407 webserver.cc:460] Webserver started at http://127.2.235.254:43017/ using document root <none> and password file <none>
I20260620 20:21:35.368058 19407 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:35.368112 19407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:35.371502 19407 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:35.373504 19422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.374578 19407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:21:35.374701 19407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "1e567951b53b4219928daf572f461847"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "985def8a1e700a752bf126e529a69bfc"
server_key_iv: "d113450d65c1371476b5b3233c8401b7"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.375168 19407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:35.390025 19407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:35.392916 19407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:35.393111 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:35.399818 19407 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:42387
I20260620 20:21:35.400080 19474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:42387 every 8 connection(s)
I20260620 20:21:35.400766 19407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:35.404636 19475 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:21:35.406152 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap starting.
I20260620 20:21:35.408406 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19407
I20260620 20:21:35.408911 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:38677
--local_ip_for_outbound_sockets=127.2.235.193
--tserver_master_addrs=127.2.235.254:42387
--webserver_port=45559
--webserver_interface=127.2.235.193
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:35.424576 19475 log.cc:824] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:35.435339 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:21:35.435664 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Bootstrap complete.
I20260620 20:21:35.439258 19475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:35.439658 19475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e567951b53b4219928daf572f461847, State: Initialized, Role: FOLLOWER
I20260620 20:21:35.440106 19475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:35.440245 19475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:35.440335 19475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:35.440450 19475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:21:35.442379 19475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:35.442692 19475 leader_election.cc:302] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e567951b53b4219928daf572f461847; no voters:
I20260620 20:21:35.443051 19475 leader_election.cc:288] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:21:35.443122 19479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:21:35.443328 19479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [term 3 LEADER]: Becoming Leader. State: Replica: 1e567951b53b4219928daf572f461847, State: Running, Role: LEADER
I20260620 20:21:35.443622 19479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } }
I20260620 20:21:35.444228 19475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:35.444491 19479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e567951b53b4219928daf572f461847. Latest consensus state: current_term: 3 leader_uuid: "1e567951b53b4219928daf572f461847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } } }
I20260620 20:21:35.444605 19479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:35.444950 19483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:35.444986 19480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1e567951b53b4219928daf572f461847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e567951b53b4219928daf572f461847" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42387 } } }
I20260620 20:21:35.445175 19480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:35.449564 19483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:35.450229 19483 catalog_manager.cc:1259] Loaded cluster ID: 833b3666744645a38fe6430578d96809
I20260620 20:21:35.450301 19483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:35.451865 19483 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:35.452455 19483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Loaded TSK: 0
I20260620 20:21:35.452541 19483 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1e567951b53b4219928daf572f461847: Loaded TSK: 1
F20260620 20:21:35.453362 19483 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4f8f9679ad google::LogMessage::Fail() at ??:0
@ 0x7f4f8f96a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4f8f9674de google::LogMessage::Flush() at ??:0
@ 0x7f4f8f968a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4f9262b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4f92623f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4f9a2cc7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4f9a2a699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4f9a2a7178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4f9a2e8df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4f9a30782b std::function<>::operator()() at ??:0
@ 0x7f4f9a2a6b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4f9a2a7f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4f9a2a4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4f9a2e8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4f91bb93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4f91bb9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4f91bbb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4f91baa512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f931ee6db start_thread at ??:0
@ 0x7f4f8e46561f clone at ??:0
W20260620 20:21:35.550544 19477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:35.550742 19477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:35.550802 19477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:35.554699 19477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:35.554749 19477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:35.554829 19477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:35.559487 19477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:38677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=45559
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:35.560417 19477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:35.561385 19477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:35.570060 19525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.570102 19522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.570097 19523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.570339 19477 server_base.cc:1034] running on GCE node
I20260620 20:21:35.570675 19477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:35.571298 19477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:35.572466 19477 hybrid_clock.cc:648] HybridClock initialized: now 1781986895572454 us; error 26 us; skew 500 ppm
Jun 20 20:21:35 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:35.575034 19477 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:21:35.575807 19477 webserver.cc:460] Webserver started at http://127.2.235.193:45559/ using document root <none> and password file <none>
I20260620 20:21:35.576295 19477 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:35.576350 19477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:35.579942 19477 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:35.582062 19532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.583663 19477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:35.583753 19477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "daeb5f1388e741248570d0a0ceeab6a2"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "38febb8697d45d2f0743d8e493d3a75c"
server_key_iv: "5de8419c6f691847ea178f40dde3537a"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.584226 19477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:35.617043 19477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:35.619998 19477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:35.620174 19477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:35.620707 19477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:35.621734 19539 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:21:35.625550 19477 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:21:35.625607 19477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 20:21:35.625667 19477 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:21:35.627184 19477 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:21:35.627234 19477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 20:21:35.627466 19539 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap starting.
I20260620 20:21:35.637776 19477 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:38677
I20260620 20:21:35.637998 19646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:38677 every 8 connection(s)
I20260620 20:21:35.638870 19477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:35.645537 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19477
I20260620 20:21:35.645964 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:44265
--local_ip_for_outbound_sockets=127.2.235.194
--tserver_master_addrs=127.2.235.254:42387
--webserver_port=44637
--webserver_interface=127.2.235.194
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:21:35.647514 19539 log.cc:824] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Log is configured to *not* fsync() on all Append() calls
W20260620 20:21:35.648655 19649 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:21:35.640978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:44897)
0620 20:21:35.641534 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.641591 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:35.642892 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:35.647788 (+ 4896us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
W20260620 20:21:35.649103 19647 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:42387 (0 consecutive failures): Network error: Failed to ping master at 127.2.235.254:42387: Client connection negotiation failed: client connection to 127.2.235.254:42387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:21:35.649813 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.649521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:36913)
0620 20:21:35.649666 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.649734 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:21:35.650579 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.650316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:39129)
0620 20:21:35.650445 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.650504 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:21:35.650803 19647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:21:35.654496 19539 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:21:35.654808 19539 tablet_bootstrap.cc:492] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Bootstrap complete.
I20260620 20:21:35.655810 19539 ts_tablet_manager.cc:1397] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.012s
I20260620 20:21:35.657862 19539 raft_consensus.cc:348] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:35.658236 19539 raft_consensus.cc:729] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: daeb5f1388e741248570d0a0ceeab6a2, State: Initialized, Role: FOLLOWER
I20260620 20:21:35.658628 19539 consensus_queue.cc:260] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:35.659442 19539 ts_tablet_manager.cc:1428] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260620 20:21:35.659675 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.659446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:60473)
0620 20:21:35.659568 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.659614 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:21:35.660249 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.660048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:40223)
0620 20:21:35.660173 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.660208 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 20:21:35.749807 19650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:35.750032 19650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:35.750098 19650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:35.753949 19650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:35.754030 19650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:35.754128 19650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:35.758647 19650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:44265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=44637
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:35.759459 19650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:35.760356 19650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:35.768675 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.768659 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.768776 19650 server_base.cc:1034] running on GCE node
W20260620 20:21:35.768658 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.769289 19650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:35.769832 19650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:35.771023 19650 hybrid_clock.cc:648] HybridClock initialized: now 1781986895770994 us; error 39 us; skew 500 ppm
Jun 20 20:21:35 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:35.773813 19650 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:21:35.774463 19650 webserver.cc:460] Webserver started at http://127.2.235.194:44637/ using document root <none> and password file <none>
I20260620 20:21:35.774878 19650 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:35.774928 19650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:35.778112 19650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:35.779845 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.780877 19650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:35.780959 19650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "5a18d029108848359ab8e28d30f79432"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "78ef2bb0b697abcb9fef2e550f1b34dc"
server_key_iv: "b19da1bd97a31714ef26656b64d513a0"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.781424 19650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:35.803421 19650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:35.806226 19650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:35.806360 19650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:35.806788 19650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:35.807418 19650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:35.807459 19650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.807492 19650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:35.807513 19650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.815913 19650 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:44265
I20260620 20:21:35.815999 19780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:44265 every 8 connection(s)
I20260620 20:21:35.816804 19650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:35.819861 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.818635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:48003)
0620 20:21:35.819195 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.819281 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":113,"threads_started":1}
W20260620 20:21:35.820219 19781 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:42387 (0 consecutive failures): Network error: Failed to ping master at 127.2.235.254:42387: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
I20260620 20:21:35.820715 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.820512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:48737)
0620 20:21:35.820619 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.820659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:21:35.821235 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.821048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:59591)
0620 20:21:35.821138 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.821181 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:21:35.821434 19781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:21:35.823231 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19650
I20260620 20:21:35.823560 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:41747
--local_ip_for_outbound_sockets=127.2.235.195
--tserver_master_addrs=127.2.235.254:42387
--webserver_port=35339
--webserver_interface=127.2.235.195
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:35.920579 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:35.920801 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:35.920867 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:35.924419 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:35.924494 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:35.924573 19784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:35.928900 19784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=35339
--tserver_master_addrs=127.2.235.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:35.929786 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:35.930650 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:35.939065 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.939049 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:35.939033 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:35.939248 19784 server_base.cc:1034] running on GCE node
I20260620 20:21:35.939762 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:35.940300 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:35.941444 19784 hybrid_clock.cc:648] HybridClock initialized: now 1781986895941419 us; error 40 us; skew 500 ppm
Jun 20 20:21:35 dist-test-slave-8z53 krb5kdc[18333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:35.943967 19784 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:21:35.944621 19784 webserver.cc:460] Webserver started at http://127.2.235.195:35339/ using document root <none> and password file <none>
I20260620 20:21:35.945045 19784 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:35.945096 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:35.948294 19784 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:35.950011 19799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.951061 19784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:35.951157 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "906a6e3ad3af455db800b52af68e3961"
format_stamp: "Formatted at 2026-06-20 20:20:53 on dist-test-slave-8z53"
server_key: "3ef775d6d406cfef59bc44d7cf81d353"
server_key_iv: "bf380be8722c48ea6d83e29b4ff6227b"
server_key_version: "encryptionkey@0"
I20260620 20:21:35.951566 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:35.962278 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:35.965034 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:35.965200 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:35.965691 19784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:35.966369 19784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:35.966416 19784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.966473 19784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:35.966507 19784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:35.974694 19784 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:41747
I20260620 20:21:35.974776 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:41747 every 8 connection(s)
I20260620 20:21:35.975524 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:21:35.978461 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19784
I20260620 20:21:35.978505 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.977193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:53581)
0620 20:21:35.977802 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.977886 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":105,"threads_started":1}
W20260620 20:21:35.978951 19913 heartbeater.cc:644] Failed to heartbeat to 127.2.235.254:42387 (0 consecutive failures): Network error: Failed to ping master at 127.2.235.254:42387: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
I20260620 20:21:35.979566 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.979335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:41677)
0620 20:21:35.979439 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.979498 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:21:35.980166 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:35.979988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:54729)
0620 20:21:35.980081 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:35.980120 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 20:21:35.980357 19913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:21:36.661214 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.660827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:56209)
0620 20:21:36.661087 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.661134 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260620 20:21:36.822234 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.821890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:46087)
0620 20:21:36.822095 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.822140 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260620 20:21:36.981096 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:36.980779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:60301)
0620 20:21:36.980972 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:36.981016 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20260620 20:21:37.246690 19920 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:37.246902 19920 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:37.247902 19920 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
Jun 20 20:21:37 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.194@KRBTEST.COM
Jun 20 20:21:37 dist-test-slave-8z53 krb5kdc[18333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.195@KRBTEST.COM
I20260620 20:21:37.260950 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.248177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:41747 (local address 127.2.235.193:49283)
0620 20:21:37.248387 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:37.248401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:37.248491 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:37.250386 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:37.250392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:37.250894 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:37.251370 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:37.251383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.252550 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.252552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:37.253558 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:37.253570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.254275 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.254281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:37.254485 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.255268 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:37.255279 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:37.256885 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:37.258726 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:37.258739 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:37.258751 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:37.259067 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:37.259381 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:37.259386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:37.259515 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:37.260075 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:37.260080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:37.260280 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 20:21:37.260435 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:21:37.260722 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":61,"thread_start_us":64,"threads_started":1}
I20260620 20:21:37.260960 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.248316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:49283 (local address 127.2.235.195:41747)
0620 20:21:37.248770 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.248800 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.248867 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:21:37.249685 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.249691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.250056 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.250327 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.251518 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.252373 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.253690 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.254140 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.254306 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.257024 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:37.257028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:37.257038 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:37.257060 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:37.258596 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259168 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:37.259172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:37.259176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:37.259285 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259621 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:37.259626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:37.259631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:37.259970 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:37.260098 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.260543 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.260797 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":120,"threads_started":1}
I20260620 20:21:37.261171 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.248678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:46727 (local address 127.2.235.194:44265)
0620 20:21:37.249008 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:37.249031 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:37.249068 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:21:37.249781 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:37.249787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:37.250094 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:37.250354 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:37.251523 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.252367 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.253760 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.254174 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.254389 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.257040 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:37.257044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:37.257053 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:37.257075 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:37.259147 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259531 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:37.259536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:37.259541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:37.259654 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259943 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:37.259947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:37.259949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:37.260279 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:37.260405 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:37.260728 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 20:21:37.260973 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":116,"threads_started":1}
I20260620 20:21:37.261590 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.248493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:44265 (local address 127.2.235.193:46727)
0620 20:21:37.248767 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:37.248781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:37.248904 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:37.250389 (+ 1485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:37.250394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:37.250895 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:37.251369 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:37.251383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.252548 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.252556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:37.253552 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:37.253561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:37.254343 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:37.254346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:37.254476 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:37.255155 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:37.255177 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:37.256586 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:37.259236 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:37.259245 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:37.259425 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:37.259775 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:37.259778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:37.259780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:37.259855 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:37.260378 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:37.260380 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:37.260460 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 20:21:37.260585 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 20:21:37.260816 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":238,"spinlock_wait_cycles":22528,"thread_start_us":51,"threads_started":1}
W20260620 20:21:37.263104 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
W20260620 20:21:37.263350 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:37.263423 19534 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeb5f1388e741248570d0a0ceeab6a2; no voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961
I20260620 20:21:37.263695 19920 raft_consensus.cc:2726] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:21:37.662202 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.661967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:41629)
0620 20:21:37.662085 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:37.662133 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:21:37.823258 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.822902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:34523)
0620 20:21:37.823132 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:37.823179 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260620 20:21:37.982089 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:37.981734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:37387)
0620 20:21:37.981969 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:37.982015 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260620 20:21:38.663422 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:38.662970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:52255)
0620 20:21:38.663276 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:38.663333 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":64,"threads_started":1}
I20260620 20:21:38.824421 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:38.823951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:53811)
0620 20:21:38.824204 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:38.824291 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260620 20:21:38.983032 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:38.982662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:36233)
0620 20:21:38.982898 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:38.982946 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260620 20:21:39.269013 19930 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:39.269131 19930 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:39.269577 19930 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
W20260620 20:21:39.270475 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
W20260620 20:21:39.270583 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:39.270617 19534 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeb5f1388e741248570d0a0ceeab6a2; no voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961
I20260620 20:21:39.270728 19930 raft_consensus.cc:2726] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:21:39.664529 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.664184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:50085)
0620 20:21:39.664394 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.664444 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260620 20:21:39.825582 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.825198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:45595)
0620 20:21:39.825463 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.825509 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":123,"threads_started":1}
I20260620 20:21:39.984438 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:39.983935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:37363)
0620 20:21:39.984260 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:39.984332 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260620 20:21:40.665823 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.665436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:47917)
0620 20:21:40.665660 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.665737 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20260620 20:21:40.826604 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.826204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:39543)
0620 20:21:40.826478 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.826528 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260620 20:21:40.985530 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:40.985114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:48259)
0620 20:21:40.985384 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:40.985432 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":68,"threads_started":1}
I20260620 20:21:41.234788 19937 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:41.234932 19937 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:41.235306 19937 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
W20260620 20:21:41.236048 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
W20260620 20:21:41.236191 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:41.236269 19534 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeb5f1388e741248570d0a0ceeab6a2; no voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961
I20260620 20:21:41.236449 19937 raft_consensus.cc:2726] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:21:41.666894 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.666504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:60295)
0620 20:21:41.666749 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:41.666802 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260620 20:21:41.827585 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.827202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:55149)
0620 20:21:41.827451 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:41.827503 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":66,"threads_started":1}
I20260620 20:21:41.986560 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:41.986231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:55293)
0620 20:21:41.986438 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:41.986485 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":64,"threads_started":1}
I20260620 20:21:42.668004 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.667624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:34121)
0620 20:21:42.667868 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:42.667920 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
I20260620 20:21:42.828630 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.828238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:57069)
0620 20:21:42.828507 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:42.828555 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":63,"threads_started":1}
I20260620 20:21:42.987704 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:42.987251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:48471)
0620 20:21:42.987573 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:42.987622 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":67,"threads_started":1}
I20260620 20:21:43.575292 19944 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:43.575476 19944 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:43.576092 19944 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
W20260620 20:21:43.576566 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
W20260620 20:21:43.576747 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:43.576813 19534 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeb5f1388e741248570d0a0ceeab6a2; no voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961
I20260620 20:21:43.576938 19944 raft_consensus.cc:2726] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:21:43.669183 19945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:43.668788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:59115)
0620 20:21:43.669047 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:43.669099 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260620 20:21:43.829720 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:43.829303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:43723)
0620 20:21:43.829572 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:43.829639 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260620 20:21:43.988765 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:43.988403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:42777)
0620 20:21:43.988633 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:43.988682 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260620 20:21:44.670502 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:44.670058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:38733)
0620 20:21:44.670329 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:44.670387 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260620 20:21:44.830662 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:44.830320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:60547)
0620 20:21:44.830525 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:44.830570 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260620 20:21:44.989832 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:44.989481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:56481)
0620 20:21:44.989703 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:44.989752 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260620 20:21:45.671677 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:45.671305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:47933)
0620 20:21:45.671522 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:45.671572 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260620 20:21:45.831713 19952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:45.831367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:38357)
0620 20:21:45.831590 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:45.831637 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260620 20:21:45.990799 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:45.990456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:35193)
0620 20:21:45.990676 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:45.990722 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260620 20:21:46.200043 19954 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:46.200201 19954 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:46.200596 19954 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
W20260620 20:21:46.201332 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
W20260620 20:21:46.201455 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:46.201490 19534 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeb5f1388e741248570d0a0ceeab6a2; no voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961
I20260620 20:21:46.201617 19954 raft_consensus.cc:2726] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:21:46.672819 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:46.672440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:36045)
0620 20:21:46.672652 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:46.672698 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260620 20:21:46.832646 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:46.832309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:40965)
0620 20:21:46.832521 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:46.832570 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
I20260620 20:21:46.991701 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:46.991388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:60909)
0620 20:21:46.991584 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:46.991629 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260620 20:21:47.673971 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:47.673517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:52557)
0620 20:21:47.673823 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:47.673879 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
I20260620 20:21:47.833719 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:47.833341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:55981)
0620 20:21:47.833574 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:47.833623 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
I20260620 20:21:47.992651 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:47.992313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:52933)
0620 20:21:47.992527 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:47.992574 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260620 20:21:48.675161 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:48.674775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:38097)
0620 20:21:48.675032 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:48.675077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
I20260620 20:21:48.834715 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:48.834359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:34477)
0620 20:21:48.834578 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:48.834630 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260620 20:21:48.993726 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:48.993373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:57389)
0620 20:21:48.993582 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:48.993631 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260620 20:21:49.676165 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:49.675814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:35497)
0620 20:21:49.676029 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:49.676078 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260620 20:21:49.835861 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:49.835434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:37473)
0620 20:21:49.835690 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:49.835760 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260620 20:21:49.994704 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:49.994319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.195:60581)
0620 20:21:49.994565 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:49.994617 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260620 20:21:50.677438 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:50.677019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.193:59295)
0620 20:21:50.677236 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:50.677348 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260620 20:21:50.836928 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:50.836566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42387 (local address 127.2.235.194:54743)
0620 20:21:50.836803 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:50.836850 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.254:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260620 20:21:50.924199 19969 raft_consensus.cc:482] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:50.924330 19969 raft_consensus.cc:504] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906a6e3ad3af455db800b52af68e3961" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 41747 } } peers { permanent_uuid: "5a18d029108848359ab8e28d30f79432" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 44265 } } peers { permanent_uuid: "daeb5f1388e741248570d0a0ceeab6a2" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38677 } }
I20260620 20:21:50.924744 19969 leader_election.cc:288] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747), 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265)
W20260620 20:21:50.925545 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a18d029108848359ab8e28d30f79432 (127.2.235.194:44265): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
W20260620 20:21:50.925691 19534 leader_election.cc:341] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 906a6e3ad3af455db800b52af68e3961 (127.2.235.195:41747): Not found: Tablet not found: 6c1ab79c6f384b7589c3ef3f7fb77e1e
I20260620 20:21:50.925770 19534 leader_election.cc:302] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeb5f1388e741248570d0a0ceeab6a2; no voters: 5a18d029108848359ab8e28d30f79432, 906a6e3ad3af455db800b52af68e3961
I20260620 20:21:50.925909 19969 raft_consensus.cc:2726] T 6c1ab79c6f384b7589c3ef3f7fb77e1e P daeb5f1388e741248570d0a0ceeab6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:21:50.983980 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 19477
I20260620 20:21:50.989598 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 19650
I20260620 20:21:50.994544 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 19784
2026-06-20T20:21:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58843 ms)
[----------] 30 tests from SecurityITest (224743 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:51 dist-test-slave-8z53 krb5kdc[19972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:51 dist-test-slave-8z53 krb5kdc[19972](info): set up 2 sockets
Jun 20 20:21:51 dist-test-slave-8z53 krb5kdc[19972](info): commencing operation
krb5kdc: starting...
W20260620 20:21:51.969132 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:21:52.074236 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44889
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:52.173924 19985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:52.174150 19985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:52.174187 19985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:52.177999 19985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:52.178042 19985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:52.178064 19985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:52.178115 19985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:52.178140 19985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:52.183044 19985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:44889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:52.183856 19985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:52.184741 19985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:52.192260 19991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:52.192260 19990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:52.192296 19993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:52.192607 19985 server_base.cc:1034] running on GCE node
I20260620 20:21:52.193148 19985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:52.194187 19985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:52.195348 19985 hybrid_clock.cc:648] HybridClock initialized: now 1781986912195331 us; error 28 us; skew 500 ppm
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:52.198024 19985 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:21:52.198869 19985 webserver.cc:460] Webserver started at http://127.2.235.254:37293/ using document root <none> and password file <none>
I20260620 20:21:52.199406 19985 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:52.199513 19985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:52.199783 19985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:52.202323 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "7a58799884cc467fa42a9b2f4d4c0b39"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "70b47f8785e3b238e67f966dcca9ff38"
server_key_iv: "3b87f3db27a375bb020959871238f3c1"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.203028 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "7a58799884cc467fa42a9b2f4d4c0b39"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "70b47f8785e3b238e67f966dcca9ff38"
server_key_iv: "3b87f3db27a375bb020959871238f3c1"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.207839 19985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:21:52.210352 20000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.211458 19985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:52.211560 19985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "7a58799884cc467fa42a9b2f4d4c0b39"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "70b47f8785e3b238e67f966dcca9ff38"
server_key_iv: "3b87f3db27a375bb020959871238f3c1"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.211637 19985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:52.221340 19985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:52.223821 19985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:52.223951 19985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:52.230239 19985 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:44889
I20260620 20:21:52.230319 20052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:44889 every 8 connection(s)
I20260620 20:21:52.231031 19985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:52.233554 20053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:52.238514 20053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39: Bootstrap starting.
I20260620 20:21:52.239109 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 19985
I20260620 20:21:52.239216 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:21:52.239502 2991 external_mini_cluster.cc:1428] Setting key 5a9e55adafc99812cc55bc47e683d512
I20260620 20:21:52.241061 20053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:52.241925 20053 log.cc:824] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:52.244306 20053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39: No bootstrap required, opened a new log
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:52.248744 20053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44889 } }
I20260620 20:21:52.248930 20053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:52.248991 20053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a58799884cc467fa42a9b2f4d4c0b39, State: Initialized, Role: FOLLOWER
I20260620 20:21:52.249432 20053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44889 } }
I20260620 20:21:52.249577 20053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:52.249671 20053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:52.249787 20053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:52.250905 20053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44889 } }
I20260620 20:21:52.251258 20053 leader_election.cc:302] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a58799884cc467fa42a9b2f4d4c0b39; no voters:
I20260620 20:21:52.251626 20053 leader_election.cc:288] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:52.251706 20058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:52.251950 20058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [term 1 LEADER]: Becoming Leader. State: Replica: 7a58799884cc467fa42a9b2f4d4c0b39, State: Running, Role: LEADER
I20260620 20:21:52.252275 20058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44889 } }
I20260620 20:21:52.252902 20053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:52.253422 20058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a58799884cc467fa42a9b2f4d4c0b39. Latest consensus state: current_term: 1 leader_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44889 } } }
I20260620 20:21:52.253544 20058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:52.253784 20058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a58799884cc467fa42a9b2f4d4c0b39" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44889 } } }
I20260620 20:21:52.253949 20062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:52.253962 20058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:52.254451 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.240626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.2.235.254:44889)
0620 20:21:52.241034 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.241055 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.241094 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:21:52.241965 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.241987 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.242381 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.242697 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:52.243548 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.244564 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.245771 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.246186 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.246300 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.249099 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:52.249105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:52.249121 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:52.249144 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:52.251157 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.251699 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.251705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.251710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.251864 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.252184 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.252187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.252188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.252617 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:52.252816 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.253117 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.253405 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":95,"threads_started":1}
I20260620 20:21:52.256996 20062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:52.261932 20062 catalog_manager.cc:1347] Generated new cluster ID: 3a6ae6500da84bc0b5392a5c6fa590bc
I20260620 20:21:52.262005 20062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:52.271144 20062 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:52.272162 20062 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:52.280561 20062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39: Generated new TSK 0
I20260620 20:21:52.281176 20062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:21:52.314316 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44889
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:52.414196 20079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:52.414443 20079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:52.414510 20079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:52.418424 20079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:52.418489 20079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:52.418561 20079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:52.423354 20079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:52.424273 20079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:52.425210 20079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:52.433597 20084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:52.433622 20085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:52.433754 20079 server_base.cc:1034] running on GCE node
W20260620 20:21:52.433893 20087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:52.434329 20079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:52.434937 20079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:52.436138 20079 hybrid_clock.cc:648] HybridClock initialized: now 1781986912436121 us; error 38 us; skew 500 ppm
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:52.438764 20079 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:21:52.439528 20079 webserver.cc:460] Webserver started at http://127.2.235.193:35613/ using document root <none> and password file <none>
I20260620 20:21:52.440033 20079 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:52.440130 20079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:52.440344 20079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:52.442255 20079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "c6e1c19dba994bc484b87c8fc64fa3a7"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "b7b19e1dcd4c2c15f995a8a1beea9968"
server_key_iv: "620b429710d26e625e0bbc25f1d1c63a"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.442782 20079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "c6e1c19dba994bc484b87c8fc64fa3a7"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "b7b19e1dcd4c2c15f995a8a1beea9968"
server_key_iv: "620b429710d26e625e0bbc25f1d1c63a"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.446092 20079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:52.448213 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.449174 20079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:52.449311 20079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "c6e1c19dba994bc484b87c8fc64fa3a7"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "b7b19e1dcd4c2c15f995a8a1beea9968"
server_key_iv: "620b429710d26e625e0bbc25f1d1c63a"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.449411 20079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:52.469420 20079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:52.472924 20079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:52.473084 20079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:52.473713 20079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:52.474421 20079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:52.474464 20079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.474500 20079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:52.474516 20079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.482384 20079 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:42635
I20260620 20:21:52.482467 20207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:42635 every 8 connection(s)
I20260620 20:21:52.483148 20079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:52.489774 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20079
I20260620 20:21:52.489920 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:21:52.490190 2991 external_mini_cluster.cc:1428] Setting key 9d9bb437e766063fd3bf828b94c0b342
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:52.496407 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.484825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:54711 (local address 127.2.235.254:44889)
0620 20:21:52.484979 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.484982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.485780 (+ 798us) server_negotiation.cc:407] Connection header received
0620 20:21:52.486963 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.486967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.487026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.487138 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:52.488155 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.488744 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.489919 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.490263 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.490371 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.492704 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:52.492710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:52.492716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:52.492746 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:52.494167 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.494792 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.494795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.494796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.494871 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.495296 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.495298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.495299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.495436 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:52.495650 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.496206 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.496304 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:21:52.497605 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.485157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44889 (local address 127.2.235.193:54711)
0620 20:21:52.485629 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:52.485667 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:52.486763 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:52.487301 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:52.487307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:52.487609 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:52.488008 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.488020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.488869 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.488873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.489765 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.489775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.490401 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.490405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.490564 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.491077 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:52.491093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:52.492527 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:52.494289 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:52.494298 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:52.494314 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:52.494683 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:52.494969 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:52.494975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:52.494980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:52.495167 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:52.495673 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:52.495680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:52.495948 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 20:21:52.496154 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 20:21:52.496558 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":113,"threads_started":1}
I20260620 20:21:52.498778 20208 heartbeater.cc:347] Connected to a master server at 127.2.235.254:44889
I20260620 20:21:52.498981 20208 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:52.499419 20208 heartbeater.cc:511] Master 127.2.235.254:44889 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
I20260620 20:21:52.500803 20017 ts_manager.cc:194] Registered new tserver with Master: c6e1c19dba994bc484b87c8fc64fa3a7 (127.2.235.193:42635)
I20260620 20:21:52.502059 20017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:54711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:21:52.514322 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44889
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:52.614552 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:52.614791 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:52.614861 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:52.618856 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:52.618924 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:52.619004 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:52.623745 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:52.624619 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:52.625566 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:52.633960 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:52.634016 20213 server_base.cc:1034] running on GCE node
W20260620 20:21:52.634053 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:52.634096 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:52.634579 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:52.635120 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:52.636384 20213 hybrid_clock.cc:648] HybridClock initialized: now 1781986912636346 us; error 44 us; skew 500 ppm
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:52.639086 20213 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:21:52.639822 20213 webserver.cc:460] Webserver started at http://127.2.235.194:42893/ using document root <none> and password file <none>
I20260620 20:21:52.640317 20213 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:52.640399 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:52.640606 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:52.642477 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "4137728718164f9c8d7c00d9294e3452"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "e5b72f70c9f881312706baa5ca9cf727"
server_key_iv: "d8246dc3621d4dd54ab53ac9bba9aa9a"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.642993 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "4137728718164f9c8d7c00d9294e3452"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "e5b72f70c9f881312706baa5ca9cf727"
server_key_iv: "d8246dc3621d4dd54ab53ac9bba9aa9a"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.646268 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:52.648298 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.649585 20213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:52.649724 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "4137728718164f9c8d7c00d9294e3452"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "e5b72f70c9f881312706baa5ca9cf727"
server_key_iv: "d8246dc3621d4dd54ab53ac9bba9aa9a"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.649834 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:52.669646 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:52.672297 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:52.672474 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:52.672972 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:52.673707 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:52.673775 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.673838 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:52.673885 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.683113 20213 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:40557
I20260620 20:21:52.683149 20341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:40557 every 8 connection(s)
I20260620 20:21:52.683928 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:52.689383 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20213
I20260620 20:21:52.689496 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:21:52.689783 2991 external_mini_cluster.cc:1428] Setting key cf9d055ae3d2ab1b0d2c908fe0b6dd0d
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:52.697690 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.685683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:43755 (local address 127.2.235.254:44889)
0620 20:21:52.685805 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.685808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.686584 (+ 776us) server_negotiation.cc:407] Connection header received
0620 20:21:52.687819 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.687822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.687876 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.687972 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:52.688956 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.689662 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.690906 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.691170 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.691255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.694041 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:52.694048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:52.694054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:52.694084 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:52.695818 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.696275 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.696282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.696286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.696368 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.696660 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.696664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.696667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.696885 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:52.696984 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.697426 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.697533 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:21:52.698288 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.686004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44889 (local address 127.2.235.194:43755)
0620 20:21:52.686440 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:52.686478 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:52.687611 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:52.688136 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:52.688142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:52.688433 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:52.688826 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.688838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.689792 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.689795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.690772 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.690782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.691320 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.691324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.691507 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.692067 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:52.692092 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:52.693871 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:52.695925 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:52.695930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:52.695939 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:52.696173 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:52.696456 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:52.696458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:52.696459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:52.696573 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:52.696980 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:52.696984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:52.697159 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 20:21:52.697361 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 20:21:52.697567 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
I20260620 20:21:52.699347 20342 heartbeater.cc:347] Connected to a master server at 127.2.235.254:44889
I20260620 20:21:52.699581 20342 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:52.699996 20342 heartbeater.cc:511] Master 127.2.235.254:44889 requested a full tablet report, sending...
I20260620 20:21:52.700995 20017 ts_manager.cc:194] Registered new tserver with Master: 4137728718164f9c8d7c00d9294e3452 (127.2.235.194:40557)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:21:52.701700 20017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:43755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:21:52.713205 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44889
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:52.809760 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:52.809949 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:52.809984 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:52.813480 20347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:52.813519 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:52.813570 20347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:52.818321 20347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:44889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:52.819085 20347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:52.819887 20347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:52.828279 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:52.828271 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:52.828271 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:52.828449 20347 server_base.cc:1034] running on GCE node
I20260620 20:21:52.828972 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:52.829541 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:52.830724 20347 hybrid_clock.cc:648] HybridClock initialized: now 1781986912830691 us; error 40 us; skew 500 ppm
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:52.833292 20347 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:21:52.834035 20347 webserver.cc:460] Webserver started at http://127.2.235.195:46317/ using document root <none> and password file <none>
I20260620 20:21:52.834532 20347 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:52.834587 20347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:52.834805 20347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:52.836581 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "96a4ab7c9677403f945b475e5ad0245c"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "dce95c43504d6150cdccc0dd81947cfc"
server_key_iv: "36457580c698c7b8ddfeecbcbe192d30"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.837114 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "96a4ab7c9677403f945b475e5ad0245c"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "dce95c43504d6150cdccc0dd81947cfc"
server_key_iv: "36457580c698c7b8ddfeecbcbe192d30"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.840401 20347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:52.842522 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.843552 20347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:52.843677 20347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "96a4ab7c9677403f945b475e5ad0245c"
format_stamp: "Formatted at 2026-06-20 20:21:52 on dist-test-slave-8z53"
server_key: "dce95c43504d6150cdccc0dd81947cfc"
server_key_iv: "36457580c698c7b8ddfeecbcbe192d30"
server_key_version: "encryptionkey@0"
I20260620 20:21:52.843765 20347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:52.871230 20347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:52.874176 20347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:52.874375 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:52.874920 20347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:52.875725 20347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:52.875771 20347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.875809 20347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:52.875847 20347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:52.885120 20347 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:39099
I20260620 20:21:52.885140 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:39099 every 8 connection(s)
I20260620 20:21:52.885949 20347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:21:52.888298 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20347
I20260620 20:21:52.888402 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:21:52.888670 2991 external_mini_cluster.cc:1428] Setting key f6c376697a674b7ae7e6eaf7abbe56d6
Jun 20 20:21:52 dist-test-slave-8z53 krb5kdc[19972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986912, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:52.898258 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.887518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:56311 (local address 127.2.235.254:44889)
0620 20:21:52.887639 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.887642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.888498 (+ 856us) server_negotiation.cc:407] Connection header received
0620 20:21:52.889807 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.889810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.889863 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.889951 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:52.890911 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.891477 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.892562 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.892770 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.892868 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.895064 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:52.895068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:52.895071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:52.895101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:52.896470 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.896944 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.896948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.896949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.897019 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.897349 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.897352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.897354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.897504 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:52.897597 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.898014 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.898111 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 20:21:52.898973 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.887886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44889 (local address 127.2.235.195:56311)
0620 20:21:52.888373 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:52.888421 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:52.889608 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:52.890101 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:52.890106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:52.890401 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:52.890785 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.890796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.891594 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.891598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.892448 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.892456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.892896 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.892900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.893051 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.893545 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:52.893558 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:52.894912 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:52.896606 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:52.896612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:52.896623 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:52.896843 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:52.897101 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:52.897105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:52.897107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:52.897226 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:52.897607 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:52.897613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:52.897808 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 20:21:52.897962 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:21:52.898214 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":138,"threads_started":1}
I20260620 20:21:52.900014 20476 heartbeater.cc:347] Connected to a master server at 127.2.235.254:44889
I20260620 20:21:52.900229 20476 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:52.900642 20476 heartbeater.cc:511] Master 127.2.235.254:44889 requested a full tablet report, sending...
I20260620 20:21:52.901582 20017 ts_manager.cc:194] Registered new tserver with Master: 96a4ab7c9677403f945b475e5ad0245c (127.2.235.195:39099)
I20260620 20:21:52.902168 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:21:52.902240 20017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:56311
I20260620 20:21:52.910295 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.904261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.2.235.254:44889)
0620 20:21:52.904366 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.904370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.904540 (+ 170us) server_negotiation.cc:407] Connection header received
0620 20:21:52.904627 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.904629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.904660 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.904714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:52.905394 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.905843 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.906565 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.906730 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.906810 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.907823 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:52.907825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:52.907827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:52.907844 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:52.909050 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.909436 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.909438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.909439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.909496 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:52.909693 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:52.909695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:52.909696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:52.909860 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:52.909945 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.910143 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.910209 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:21:52.912747 20017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:21:52.915870 20017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:21:52.930272 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.924528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:39099 (local address 127.0.0.1:50854)
0620 20:21:52.925062 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:52.925075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:52.925165 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:52.925938 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:52.925943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:52.925964 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:52.926074 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.926080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.927250 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.927252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.928553 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.928562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.929738 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.929742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.929852 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.929897 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 20:21:52.930001 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:21:52.930109 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":124,"thread_start_us":82,"threads_started":1}
I20260620 20:21:52.930416 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.924405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40557 (local address 127.0.0.1:55268)
0620 20:21:52.924898 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:52.924946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:52.925067 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:52.926022 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:52.926025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:52.926041 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:52.926159 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.926167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.927591 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.927594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.928849 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.928859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.929993 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.929997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.930098 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.930128 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:21:52.930199 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:21:52.930275 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":110,"thread_start_us":90,"threads_started":1}
I20260620 20:21:52.930696 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.925112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.2.235.195:39099)
0620 20:21:52.925476 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.925487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.925506 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:21:52.925566 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.925570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.925776 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.925935 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:52.926191 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.927151 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.928665 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.929415 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.929524 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.929655 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.930457 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.930565 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260620 20:21:52.931051 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.924509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.2.235.194:40557)
0620 20:21:52.925514 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.925526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.925549 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:21:52.925629 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.925636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.925836 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.926315 (+ 479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:52.926488 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.927456 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.928973 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.929860 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.929993 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.930127 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.930232 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.930652 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":134,"threads_started":1}
I20260620 20:21:52.931094 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.925505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.2.235.193:42635)
0620 20:21:52.926251 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:52.926266 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:52.926516 (+ 250us) server_negotiation.cc:407] Connection header received
0620 20:21:52.926621 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:52.926626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:52.926785 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:52.926924 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:52.927179 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.928113 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.929449 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.930038 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.930134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.930274 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:52.930765 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 20:21:52.930955 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":114,"threads_started":1}
I20260620 20:21:52.933367 20407 tablet_service.cc:1467] Processing CreateTablet for tablet 5d971f7c63d04bb9a40ac2ecf19980d8 (DEFAULT_TABLE table=test-table [id=80cf958d69084f6d910d666f77d8228d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:52.933681 20276 tablet_service.cc:1467] Processing CreateTablet for tablet 5d971f7c63d04bb9a40ac2ecf19980d8 (DEFAULT_TABLE table=test-table [id=80cf958d69084f6d910d666f77d8228d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:52.934342 20407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d971f7c63d04bb9a40ac2ecf19980d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:52.934576 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d971f7c63d04bb9a40ac2ecf19980d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:52.940096 20494 tablet_bootstrap.cc:492] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Bootstrap starting.
I20260620 20:21:52.940097 20493 tablet_bootstrap.cc:492] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Bootstrap starting.
I20260620 20:21:52.941037 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:52.925759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:42635 (local address 127.0.0.1:44500)
0620 20:21:52.926392 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:52.926405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:52.926512 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:52.926933 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:52.926936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:52.926955 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:52.927063 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.927069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.928280 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.928283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.929333 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:52.929342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:52.930331 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:52.930334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:52.930625 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:52.930653 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:21:52.930742 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:21:52.940919 (+ 10177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":75,"thread_start_us":117,"threads_started":1}
I20260620 20:21:52.942631 20493 tablet_bootstrap.cc:654] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:52.942938 20494 tablet_bootstrap.cc:654] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:52.943440 20493 log.cc:824] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:52.943560 20494 log.cc:824] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:52.943843 20142 tablet_service.cc:1467] Processing CreateTablet for tablet 5d971f7c63d04bb9a40ac2ecf19980d8 (DEFAULT_TABLE table=test-table [id=80cf958d69084f6d910d666f77d8228d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:52.944805 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d971f7c63d04bb9a40ac2ecf19980d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:52.945463 20493 tablet_bootstrap.cc:492] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: No bootstrap required, opened a new log
I20260620 20:21:52.945463 20494 tablet_bootstrap.cc:492] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: No bootstrap required, opened a new log
I20260620 20:21:52.945634 20493 ts_tablet_manager.cc:1397] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 20:21:52.945636 20494 ts_tablet_manager.cc:1397] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:21:52.949069 20493 raft_consensus.cc:348] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:52.949069 20494 raft_consensus.cc:348] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:52.949332 20494 raft_consensus.cc:374] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:52.949332 20493 raft_consensus.cc:374] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:52.949388 20493 raft_consensus.cc:729] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4137728718164f9c8d7c00d9294e3452, State: Initialized, Role: FOLLOWER
I20260620 20:21:52.949388 20494 raft_consensus.cc:729] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a4ab7c9677403f945b475e5ad0245c, State: Initialized, Role: FOLLOWER
I20260620 20:21:52.949782 20493 consensus_queue.cc:260] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:52.949782 20494 consensus_queue.cc:260] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:52.950394 20342 heartbeater.cc:503] Master 127.2.235.254:44889 was elected leader, sending a full tablet report...
I20260620 20:21:52.950651 20493 ts_tablet_manager.cc:1428] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 20:21:52.950661 20476 heartbeater.cc:503] Master 127.2.235.254:44889 was elected leader, sending a full tablet report...
I20260620 20:21:52.951367 20494 ts_tablet_manager.cc:1428] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:21:52.951752 20499 tablet_bootstrap.cc:492] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: Bootstrap starting.
I20260620 20:21:52.953745 20499 tablet_bootstrap.cc:654] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:52.955003 20499 log.cc:824] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:52.956579 20499 tablet_bootstrap.cc:492] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: No bootstrap required, opened a new log
I20260620 20:21:52.956861 20499 ts_tablet_manager.cc:1397] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:52.959405 20499 raft_consensus.cc:348] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:52.959612 20499 raft_consensus.cc:374] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:52.959689 20499 raft_consensus.cc:729] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e1c19dba994bc484b87c8fc64fa3a7, State: Initialized, Role: FOLLOWER
I20260620 20:21:52.960028 20499 consensus_queue.cc:260] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:52.960582 20208 heartbeater.cc:503] Master 127.2.235.254:44889 was elected leader, sending a full tablet report...
I20260620 20:21:52.960759 20499 ts_tablet_manager.cc:1428] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260620 20:21:52.984112 20209 tablet.cc:2369] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:53.093334 20497 raft_consensus.cc:482] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:53.093538 20497 raft_consensus.cc:504] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:53.094739 20497 leader_election.cc:288] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4137728718164f9c8d7c00d9294e3452 (127.2.235.194:40557), c6e1c19dba994bc484b87c8fc64fa3a7 (127.2.235.193:42635)
I20260620 20:21:53.098766 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.095188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:58419 (local address 127.2.235.193:42635)
0620 20:21:53.095332 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.095336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.095359 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:21:53.095404 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.095406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.095444 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.095552 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:53.095829 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.096372 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.097942 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.098279 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.098363 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.098397 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.098584 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.098655 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:21:53.098767 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.094997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:42635 (local address 127.2.235.195:58419)
0620 20:21:53.095097 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.095113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.095198 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.095565 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.095568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.095586 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:53.095708 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.095719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.096478 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.096482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.097838 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.097844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.098374 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.098376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.098453 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.098471 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.098565 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.098634 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":129}
I20260620 20:21:53.099341 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.094998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:40557 (local address 127.2.235.195:54287)
0620 20:21:53.095447 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.095456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.095565 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.095855 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.095858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.095874 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:53.095983 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.095989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.096580 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.096583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.097829 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.097840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.099021 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.099024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.099111 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.099131 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.099193 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.099237 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":183,"thread_start_us":157,"threads_started":1}
I20260620 20:21:53.099411 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.095113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:54287 (local address 127.2.235.194:40557)
0620 20:21:53.095268 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.095273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.095544 (+ 271us) server_negotiation.cc:407] Connection header received
0620 20:21:53.095671 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.095674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.095712 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.095813 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:53.096084 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.096494 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.098440 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.098787 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.098912 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.098958 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.099225 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.099300 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:21:53.099388 20162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d971f7c63d04bb9a40ac2ecf19980d8" candidate_uuid: "96a4ab7c9677403f945b475e5ad0245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" is_pre_election: true
I20260620 20:21:53.099651 20162 raft_consensus.cc:2454] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a4ab7c9677403f945b475e5ad0245c in term 0.
I20260620 20:21:53.099965 20296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d971f7c63d04bb9a40ac2ecf19980d8" candidate_uuid: "96a4ab7c9677403f945b475e5ad0245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4137728718164f9c8d7c00d9294e3452" is_pre_election: true
I20260620 20:21:53.100168 20365 leader_election.cc:302] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96a4ab7c9677403f945b475e5ad0245c, c6e1c19dba994bc484b87c8fc64fa3a7; no voters:
I20260620 20:21:53.100226 20296 raft_consensus.cc:2454] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a4ab7c9677403f945b475e5ad0245c in term 0.
I20260620 20:21:53.100471 20497 raft_consensus.cc:2781] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:53.100569 20497 raft_consensus.cc:482] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:53.100629 20497 raft_consensus.cc:3037] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:53.101679 20497 raft_consensus.cc:504] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:53.102027 20497 leader_election.cc:288] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [CANDIDATE]: Term 1 election: Requested vote from peers 4137728718164f9c8d7c00d9294e3452 (127.2.235.194:40557), c6e1c19dba994bc484b87c8fc64fa3a7 (127.2.235.193:42635)
I20260620 20:21:53.102303 20296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d971f7c63d04bb9a40ac2ecf19980d8" candidate_uuid: "96a4ab7c9677403f945b475e5ad0245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4137728718164f9c8d7c00d9294e3452"
I20260620 20:21:53.102362 20162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d971f7c63d04bb9a40ac2ecf19980d8" candidate_uuid: "96a4ab7c9677403f945b475e5ad0245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7"
I20260620 20:21:53.102416 20296 raft_consensus.cc:3037] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:53.102486 20162 raft_consensus.cc:3037] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:53.103438 20296 raft_consensus.cc:2454] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a4ab7c9677403f945b475e5ad0245c in term 1.
I20260620 20:21:53.103580 20162 raft_consensus.cc:2454] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a4ab7c9677403f945b475e5ad0245c in term 1.
I20260620 20:21:53.103802 20366 leader_election.cc:302] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4137728718164f9c8d7c00d9294e3452, 96a4ab7c9677403f945b475e5ad0245c; no voters:
I20260620 20:21:53.104048 20497 raft_consensus.cc:2781] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:53.104252 20497 raft_consensus.cc:686] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 1 LEADER]: Becoming Leader. State: Replica: 96a4ab7c9677403f945b475e5ad0245c, State: Running, Role: LEADER
I20260620 20:21:53.104563 20497 consensus_queue.cc:237] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } }
I20260620 20:21:53.107676 20016 catalog_manager.cc:5495] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c reported cstate change: term changed from 0 to 1, leader changed from <none> to 96a4ab7c9677403f945b475e5ad0245c (127.2.235.195). New cstate: current_term: 1 leader_uuid: "96a4ab7c9677403f945b475e5ad0245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96a4ab7c9677403f945b475e5ad0245c" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 } health_report { overall_health: UNKNOWN } } }
W20260620 20:21:53.136821 20477 tablet.cc:2369] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:53.162122 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.158841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.2.235.195:39099)
0620 20:21:53.158979 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.158983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.159073 (+ 90us) server_negotiation.cc:407] Connection header received
0620 20:21:53.159231 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.159234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.159284 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.159387 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:53.159621 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.160092 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.160829 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.161017 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.161111 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.161422 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:53.161776 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:53.161898 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.161955 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.162003 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:21:53.167235 20162 raft_consensus.cc:1261] T 5d971f7c63d04bb9a40ac2ecf19980d8 P c6e1c19dba994bc484b87c8fc64fa3a7 [term 1 FOLLOWER]: Refusing update from remote peer 96a4ab7c9677403f945b475e5ad0245c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:53.167235 20296 raft_consensus.cc:1261] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 1 FOLLOWER]: Refusing update from remote peer 96a4ab7c9677403f945b475e5ad0245c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:53.167904 20503 consensus_queue.cc:1035] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4137728718164f9c8d7c00d9294e3452" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:53.168121 20497 consensus_queue.cc:1035] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e1c19dba994bc484b87c8fc64fa3a7" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:53.174356 20506 mvcc.cc:205] Tried to move back new op lower bound from 7299018396326952960 to 7299018396079214592. Current Snapshot: MvccSnapshot[applied={T|T < 7299018396326952960}]
I20260620 20:21:53.175233 20507 mvcc.cc:205] Tried to move back new op lower bound from 7299018396326952960 to 7299018396079214592. Current Snapshot: MvccSnapshot[applied={T|T < 7299018396326952960}]
I20260620 20:21:53.183552 20015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:53.183696 20015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:53.185933 20015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a58799884cc467fa42a9b2f4d4c0b39: Sending DeleteTablet for 3 replicas of tablet 5d971f7c63d04bb9a40ac2ecf19980d8
I20260620 20:21:53.186573 20276 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d971f7c63d04bb9a40ac2ecf19980d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:53 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:55268
I20260620 20:21:53.186654 20407 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d971f7c63d04bb9a40ac2ecf19980d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:53 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:50854
I20260620 20:21:53.186820 20142 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d971f7c63d04bb9a40ac2ecf19980d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:53 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:44500
I20260620 20:21:53.187064 20517 tablet_replica.cc:330] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: stopping tablet replica
I20260620 20:21:53.187330 20516 tablet_replica.cc:330] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: stopping tablet replica
I20260620 20:21:53.187479 20517 raft_consensus.cc:2229] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:21:53.187685 20516 raft_consensus.cc:2229] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:53.187772 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20079
I20260620 20:21:53.187846 20517 raft_consensus.cc:2258] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:53.188083 20516 raft_consensus.cc:2258] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:53.189589 20516 ts_tablet_manager.cc:1905] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:53.192185 20517 ts_tablet_manager.cc:1905] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:53.192369 20516 ts_tablet_manager.cc:1918] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:53.192454 20516 log.cc:1196] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/5d971f7c63d04bb9a40ac2ecf19980d8
I20260620 20:21:53.192742 20516 ts_tablet_manager.cc:1939] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 4137728718164f9c8d7c00d9294e3452: Deleting consensus metadata
I20260620 20:21:53.193810 20004 catalog_manager.cc:4841] TS 4137728718164f9c8d7c00d9294e3452 (127.2.235.194:40557): tablet 5d971f7c63d04bb9a40ac2ecf19980d8 (table test-table [id=80cf958d69084f6d910d666f77d8228d]) successfully deleted
I20260620 20:21:53.194820 20517 ts_tablet_manager.cc:1918] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:53.194921 20517 log.cc:1196] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/5d971f7c63d04bb9a40ac2ecf19980d8
W20260620 20:21:53.195322 20003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:42635 (error 108)
I20260620 20:21:53.195497 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20213
I20260620 20:21:53.196557 20517 ts_tablet_manager.cc:1939] T 5d971f7c63d04bb9a40ac2ecf19980d8 P 96a4ab7c9677403f945b475e5ad0245c: Deleting consensus metadata
I20260620 20:21:53.197432 20001 catalog_manager.cc:4841] TS 96a4ab7c9677403f945b475e5ad0245c (127.2.235.195:39099): tablet 5d971f7c63d04bb9a40ac2ecf19980d8 (table test-table [id=80cf958d69084f6d910d666f77d8228d]) successfully deleted
I20260620 20:21:53.197947 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.197637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:42635 (local address 127.0.0.1:44514)
0620 20:21:53.197773 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.197862 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:42635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:21:53.198094 20003 catalog_manager.cc:4584] TS c6e1c19dba994bc484b87c8fc64fa3a7 (127.2.235.193:42635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d971f7c63d04bb9a40ac2ecf19980d8: Network error: Client connection negotiation failed: client connection to 127.2.235.193:42635: connect: Connection refused (error 111)
I20260620 20:21:53.200836 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20347
I20260620 20:21:53.206523 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 19985
2026-06-20T20:21:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2211 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T20:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:53Z Disabled control of system clock
I20260620 20:21:53.234367 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42225
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:42225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:21:53.336875 20523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:53.337077 20523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:53.337109 20523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:53.340937 20523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:53.340984 20523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:53.341001 20523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:53.341019 20523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:53.341034 20523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:53.345536 20523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:42225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:42225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:53.346306 20523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:53.347406 20523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:53.355123 20528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.355170 20531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:53.355208 20523 server_base.cc:1034] running on GCE node
W20260620 20:21:53.355124 20529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:53.355975 20523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:53.356925 20523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:53.358112 20523 hybrid_clock.cc:648] HybridClock initialized: now 1781986913358067 us; error 57 us; skew 500 ppm
I20260620 20:21:53.359843 20523 webserver.cc:460] Webserver started at http://127.2.235.254:35167/ using document root <none> and password file <none>
I20260620 20:21:53.360368 20523 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:53.360452 20523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:53.360666 20523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:53.362547 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "9ed8ee2e886a40ef13ea4e2ae8e74aa5"
server_key_iv: "50c41e9f6b6f5679c237c356c080b6df"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.363093 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "9ed8ee2e886a40ef13ea4e2ae8e74aa5"
server_key_iv: "50c41e9f6b6f5679c237c356c080b6df"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.366392 20523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:53.368590 20537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.369740 20523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:53.369870 20523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "9ed8ee2e886a40ef13ea4e2ae8e74aa5"
server_key_iv: "50c41e9f6b6f5679c237c356c080b6df"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.369971 20523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:53.379864 20523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:53.380398 20523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:53.380573 20523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:53.387022 20523 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:42225
I20260620 20:21:53.387029 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:42225 every 8 connection(s)
I20260620 20:21:53.387806 20523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:53.388938 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20523
I20260620 20:21:53.389057 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:21:53.389377 2991 external_mini_cluster.cc:1428] Setting key b4f2c404a2406ac539c06400c2cd608f
I20260620 20:21:53.391054 20590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:53.396648 20590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8: Bootstrap starting.
I20260620 20:21:53.397955 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.390600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.2.235.254:42225)
0620 20:21:53.391183 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.391198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.391231 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:21:53.391794 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.391809 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.392139 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.392382 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:53.392689 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.393648 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.394852 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.395293 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.395407 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.396230 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:53.396235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:53.396254 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:53.396683 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:53.396791 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:53.396812 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:53.396883 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.397032 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.397317 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":126,"threads_started":1}
I20260620 20:21:53.398866 20590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:53.399638 20590 log.cc:824] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:53.401228 20590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8: No bootstrap required, opened a new log
I20260620 20:21:53.404372 20590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42225 } }
I20260620 20:21:53.404548 20590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:53.404632 20590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7f9aa59f7f4fd0a62ad66294a453b8, State: Initialized, Role: FOLLOWER
I20260620 20:21:53.405019 20590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42225 } }
I20260620 20:21:53.405153 20590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:53.405215 20590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:53.405364 20590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:53.406095 20590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42225 } }
I20260620 20:21:53.406423 20590 leader_election.cc:302] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b7f9aa59f7f4fd0a62ad66294a453b8; no voters:
I20260620 20:21:53.406721 20590 leader_election.cc:288] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:53.406826 20595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:53.407047 20595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5b7f9aa59f7f4fd0a62ad66294a453b8, State: Running, Role: LEADER
I20260620 20:21:53.407317 20595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42225 } }
I20260620 20:21:53.407747 20590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:53.408619 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7f9aa59f7f4fd0a62ad66294a453b8. Latest consensus state: current_term: 1 leader_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42225 } } }
I20260620 20:21:53.408748 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:53.409040 20596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7f9aa59f7f4fd0a62ad66294a453b8" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 42225 } } }
I20260620 20:21:53.409128 20596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:53.409832 20604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:53.412071 20604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:53.416883 20604 catalog_manager.cc:1347] Generated new cluster ID: 42d50b5f16004f228d9273b87ac5d9c1
I20260620 20:21:53.416949 20604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:53.423382 20604 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:53.424214 20604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:53.431066 20604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8: Generated new TSK 0
I20260620 20:21:53.431702 20604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:53.438943 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42225
--builtin_ntp_servers=127.2.235.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:21:53.540195 20614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:53.540401 20614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:53.540433 20614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:53.543972 20614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:53.544014 20614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:53.544098 20614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:53.548883 20614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:53.549782 20614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:53.550977 20614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:53.559705 20619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.559705 20620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.559713 20622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:53.559950 20614 server_base.cc:1034] running on GCE node
I20260620 20:21:53.560400 20614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:53.560949 20614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:53.562098 20614 hybrid_clock.cc:648] HybridClock initialized: now 1781986913562081 us; error 27 us; skew 500 ppm
I20260620 20:21:53.563695 20614 webserver.cc:460] Webserver started at http://127.2.235.193:45247/ using document root <none> and password file <none>
I20260620 20:21:53.564250 20614 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:53.564314 20614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:53.564465 20614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:53.566373 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "a12e6a8af7e051966e79531ea3abc1ef"
server_key_iv: "7feb5655c7c7e11b3bcb85eede1debab"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.566830 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "a12e6a8af7e051966e79531ea3abc1ef"
server_key_iv: "7feb5655c7c7e11b3bcb85eede1debab"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.570012 20614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:53.572089 20628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.573314 20614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:53.573465 20614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "a12e6a8af7e051966e79531ea3abc1ef"
server_key_iv: "7feb5655c7c7e11b3bcb85eede1debab"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.573580 20614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:53.591719 20614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:53.592195 20614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:53.592334 20614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:53.592762 20614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:53.593524 20614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:53.593568 20614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.593601 20614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:53.593622 20614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.602092 20614 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:41221
I20260620 20:21:53.602165 20742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:41221 every 8 connection(s)
I20260620 20:21:53.602907 20614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:53.604532 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20614
I20260620 20:21:53.604638 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:21:53.604943 2991 external_mini_cluster.cc:1428] Setting key 8b0440a0ddca7bbc445379348981ebc5
I20260620 20:21:53.606900 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42225
--builtin_ntp_servers=127.2.235.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:21:53.612861 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.604689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:32789 (local address 127.2.235.254:42225)
0620 20:21:53.604832 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.604836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.606079 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 20:21:53.607563 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.607566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.607619 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.607699 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:53.608713 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.609441 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.610671 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.611113 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.611259 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.612262 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:53.612268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:53.612273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:53.612291 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:53.612316 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:53.612330 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:53.612417 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.612562 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.612663 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:21:53.613554 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.605111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42225 (local address 127.2.235.193:32789)
0620 20:21:53.605922 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.605983 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.607372 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.607868 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.607875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.608051 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:53.608570 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.608583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.609565 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.609567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.610527 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.610540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.611249 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.611252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.611391 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.612000 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:21:53.612002 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:53.612040 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:21:53.612095 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:53.612421 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:53.612427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:53.612431 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.612520 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.612717 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":170,"threads_started":1}
I20260620 20:21:53.614723 20743 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42225
I20260620 20:21:53.614979 20743 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:53.615612 20743 heartbeater.cc:511] Master 127.2.235.254:42225 requested a full tablet report, sending...
I20260620 20:21:53.617081 20554 ts_manager.cc:194] Registered new tserver with Master: ad6a5d959ff04da3b80b587a1ebf4a7e (127.2.235.193:41221)
I20260620 20:21:53.618197 20554 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.193:32789
W20260620 20:21:53.713116 20746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:53.713405 20746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:53.713449 20746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:53.717386 20746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:53.717432 20746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:53.717519 20746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:53.721828 20746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:53.722641 20746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:53.723788 20746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:53.732786 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.732769 20751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.732734 20752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:53.733100 20746 server_base.cc:1034] running on GCE node
I20260620 20:21:53.733486 20746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:53.733971 20746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:53.735122 20746 hybrid_clock.cc:648] HybridClock initialized: now 1781986913735106 us; error 45 us; skew 500 ppm
I20260620 20:21:53.736698 20746 webserver.cc:460] Webserver started at http://127.2.235.194:43097/ using document root <none> and password file <none>
I20260620 20:21:53.737123 20746 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:53.737174 20746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:53.737386 20746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:53.739066 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "a23f9997046f41d6b3ba2c673db7c398"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "33b8b6e7b747e7d644b5d67cade9b0f9"
server_key_iv: "2fff49e4473f231ae721cc501f729bb0"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.739495 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "a23f9997046f41d6b3ba2c673db7c398"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "33b8b6e7b747e7d644b5d67cade9b0f9"
server_key_iv: "2fff49e4473f231ae721cc501f729bb0"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.742748 20746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:53.744781 20760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.745944 20746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:53.746029 20746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "a23f9997046f41d6b3ba2c673db7c398"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "33b8b6e7b747e7d644b5d67cade9b0f9"
server_key_iv: "2fff49e4473f231ae721cc501f729bb0"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.746098 20746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:53.762732 20746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:53.763171 20746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:53.763290 20746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:53.763686 20746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:53.764413 20746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:53.764456 20746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.764490 20746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:53.764506 20746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.771850 20746 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:39711
I20260620 20:21:53.771870 20874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:39711 every 8 connection(s)
I20260620 20:21:53.772599 20746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:53.780856 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.774171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:51789 (local address 127.2.235.254:42225)
0620 20:21:53.774303 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.774307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.775255 (+ 948us) server_negotiation.cc:407] Connection header received
0620 20:21:53.776639 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.776643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.776703 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.776799 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:53.777664 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.778158 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.779172 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.779392 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.779502 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.780383 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:53.780387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:53.780389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:53.780400 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:53.780426 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:53.780434 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:53.780505 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.780645 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.780704 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:21:53.781569 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.774561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42225 (local address 127.2.235.194:51789)
0620 20:21:53.775126 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.775160 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.776449 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.776978 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.776984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.777054 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:53.777533 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.777544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.778295 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.778298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.779056 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.779064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.779530 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.779534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.779690 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.780151 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:21:53.780152 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:53.780184 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:21:53.780234 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:53.780507 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:53.780517 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:53.780525 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.780643 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.780860 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260620 20:21:53.782601 20875 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42225
I20260620 20:21:53.782737 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20746
I20260620 20:21:53.782841 20875 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:53.782861 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:21:53.783097 2991 external_mini_cluster.cc:1428] Setting key 19929ccd9d6dcdfc6e9ffc5687c39ad3
I20260620 20:21:53.783330 20875 heartbeater.cc:511] Master 127.2.235.254:42225 requested a full tablet report, sending...
I20260620 20:21:53.784375 20553 ts_manager.cc:194] Registered new tserver with Master: a23f9997046f41d6b3ba2c673db7c398 (127.2.235.194:39711)
I20260620 20:21:53.785001 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42225
--builtin_ntp_servers=127.2.235.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:21:53.785115 20553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.194:51789
W20260620 20:21:53.885495 20878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:53.885783 20878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:53.885854 20878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:53.889696 20878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:53.889763 20878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:53.889853 20878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:53.893936 20878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:42225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:53.894838 20878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:53.896057 20878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:53.904891 20883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.904924 20884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:53.904893 20886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:53.905095 20878 server_base.cc:1034] running on GCE node
I20260620 20:21:53.905531 20878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:53.906081 20878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:53.907241 20878 hybrid_clock.cc:648] HybridClock initialized: now 1781986913907227 us; error 22 us; skew 500 ppm
I20260620 20:21:53.909039 20878 webserver.cc:460] Webserver started at http://127.2.235.195:34691/ using document root <none> and password file <none>
I20260620 20:21:53.909572 20878 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:53.909632 20878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:53.909868 20878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:53.911581 20878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "097c35c0ed47211ee66099598c1aa635"
server_key_iv: "cc910a07b54280e0e79eba26791023a8"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.912181 20878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "097c35c0ed47211ee66099598c1aa635"
server_key_iv: "cc910a07b54280e0e79eba26791023a8"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.915602 20878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:21:53.917858 20892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.918889 20878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:21:53.919003 20878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b"
format_stamp: "Formatted at 2026-06-20 20:21:53 on dist-test-slave-8z53"
server_key: "097c35c0ed47211ee66099598c1aa635"
server_key_iv: "cc910a07b54280e0e79eba26791023a8"
server_key_version: "encryptionkey@0"
I20260620 20:21:53.919099 20878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:53.928946 20878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:53.929584 20878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:53.929775 20878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:53.930349 20878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:53.931268 20878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:53.931349 20878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.931414 20878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:53.931457 20878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:53.940716 20878 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:40855
I20260620 20:21:53.940759 21007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:40855 every 8 connection(s)
I20260620 20:21:53.941545 20878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:21:53.949707 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.943101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:60231 (local address 127.2.235.254:42225)
0620 20:21:53.943221 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.943224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.944101 (+ 877us) server_negotiation.cc:407] Connection header received
0620 20:21:53.945506 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.945509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.945564 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.945630 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:53.946406 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.946906 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.947928 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.948143 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.948254 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.949156 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:53.949160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:53.949163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:53.949174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:53.949207 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:53.949214 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:53.949341 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.949496 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.949555 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:21:53.950440 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.943415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:42225 (local address 127.2.235.195:60231)
0620 20:21:53.943969 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.944002 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.945232 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.945781 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.945787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.945858 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:53.946283 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.946293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.947034 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.947039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.947815 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.947824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.948274 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.948278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.948421 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.948907 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:21:53.948909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:53.948953 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:21:53.949002 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:53.949342 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:53.949353 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:53.949360 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.949480 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.949709 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":88,"threads_started":1}
I20260620 20:21:53.950579 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 20878
I20260620 20:21:53.950670 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:21:53.950949 2991 external_mini_cluster.cc:1428] Setting key 23561feac76d0b34cc4ab373a6308c1f
I20260620 20:21:53.951526 21008 heartbeater.cc:347] Connected to a master server at 127.2.235.254:42225
I20260620 20:21:53.951726 21008 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:53.952219 21008 heartbeater.cc:511] Master 127.2.235.254:42225 requested a full tablet report, sending...
I20260620 20:21:53.953126 20553 ts_manager.cc:194] Registered new tserver with Master: d0b7b6ef0be04e44bc77ca6c59370c0b (127.2.235.195:40855)
I20260620 20:21:53.953655 20553 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.195:60231
I20260620 20:21:53.964255 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:21:53.969221 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.966061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.2.235.254:42225)
0620 20:21:53.966157 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.966160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.966309 (+ 149us) server_negotiation.cc:407] Connection header received
0620 20:21:53.966404 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.966406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.966434 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.966492 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:53.966707 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.967144 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.967880 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.968052 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.968115 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.968774 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:53.968777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:53.968778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:53.968787 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:53.968800 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:53.968830 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:53.968891 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.969054 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.969098 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:21:53.971958 20553 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:21:53.974917 20553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:21:53.987700 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.983054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.2.235.195:40855)
0620 20:21:53.983525 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.983535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.983555 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:21:53.983671 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.983676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.983835 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.983968 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:53.984359 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.985040 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.986390 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.987087 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.987203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.987376 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.987472 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.987549 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":122,"threads_started":1}
I20260620 20:21:53.987692 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.982593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40855 (local address 127.0.0.1:35616)
0620 20:21:53.983472 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.983485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.983563 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.984102 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.984106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.984130 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:53.984242 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.984249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.985153 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.985157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.986143 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.986152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.987205 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.987208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.987309 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.987343 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.987447 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.987554 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":21,"thread_start_us":86,"threads_started":1}
I20260620 20:21:53.988194 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.982422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:39711 (local address 127.0.0.1:41198)
0620 20:21:53.982700 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.982767 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.982887 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.983771 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.983777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.983802 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:53.983930 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.983937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.985395 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.985399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.986566 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.986579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.987813 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.987818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.987895 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.987918 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.987992 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.988055 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":28,"thread_start_us":58,"threads_started":1}
I20260620 20:21:53.988358 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.982559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.2.235.194:39711)
0620 20:21:53.983058 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.983068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.983088 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:21:53.983178 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.983183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.983410 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.983572 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:53.984085 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.985240 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.986700 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.987467 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.987573 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.987753 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.988024 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.988129 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":118,"threads_started":1}
I20260620 20:21:53.989068 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.983525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:41221 (local address 127.0.0.1:39082)
0620 20:21:53.984474 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:53.984489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:53.984584 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:53.985287 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:53.985290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:53.985308 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:53.985386 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.985391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.986641 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.986645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.987714 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:53.987725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.988737 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.988741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:53.988826 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.988847 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:21:53.988907 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 20:21:53.988965 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":90,"threads_started":1}
I20260620 20:21:53.989169 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:53.984044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.2.235.193:41221)
0620 20:21:53.984832 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:53.984844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:53.984868 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:21:53.984941 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:53.984946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:53.985111 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:53.985310 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:53.985466 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.986509 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.987850 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:53.988597 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:53.988724 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:53.988879 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:53.988928 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 20:21:53.989005 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":162,"threads_started":1}
I20260620 20:21:53.990681 20942 tablet_service.cc:1467] Processing CreateTablet for tablet 00b2148c1a874bfa9dbc71e5b67881a9 (DEFAULT_TABLE table=test-table [id=ae7bbe194e27438188726afe7ce4a77d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:53.991037 20809 tablet_service.cc:1467] Processing CreateTablet for tablet 00b2148c1a874bfa9dbc71e5b67881a9 (DEFAULT_TABLE table=test-table [id=ae7bbe194e27438188726afe7ce4a77d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:53.991514 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b2148c1a874bfa9dbc71e5b67881a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:53.991573 20676 tablet_service.cc:1467] Processing CreateTablet for tablet 00b2148c1a874bfa9dbc71e5b67881a9 (DEFAULT_TABLE table=test-table [id=ae7bbe194e27438188726afe7ce4a77d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:53.991997 20809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b2148c1a874bfa9dbc71e5b67881a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:53.992503 20676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b2148c1a874bfa9dbc71e5b67881a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:53.997195 21025 tablet_bootstrap.cc:492] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Bootstrap starting.
I20260620 20:21:53.998795 21026 tablet_bootstrap.cc:492] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Bootstrap starting.
I20260620 20:21:53.999723 21027 tablet_bootstrap.cc:492] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: Bootstrap starting.
I20260620 20:21:53.999855 21025 tablet_bootstrap.cc:654] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:54.000542 21026 tablet_bootstrap.cc:654] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:54.000566 21025 log.cc:824] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:54.001217 21026 log.cc:824] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:54.001405 21027 tablet_bootstrap.cc:654] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:54.002161 21027 log.cc:824] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:54.002386 21025 tablet_bootstrap.cc:492] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: No bootstrap required, opened a new log
I20260620 20:21:54.002550 21025 ts_tablet_manager.cc:1397] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 20:21:54.003044 21026 tablet_bootstrap.cc:492] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: No bootstrap required, opened a new log
I20260620 20:21:54.003337 21026 ts_tablet_manager.cc:1397] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:54.003942 21027 tablet_bootstrap.cc:492] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: No bootstrap required, opened a new log
I20260620 20:21:54.004145 21027 ts_tablet_manager.cc:1397] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 20:21:54.006806 21025 raft_consensus.cc:348] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.007046 21025 raft_consensus.cc:374] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:54.007100 21025 raft_consensus.cc:729] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7b6ef0be04e44bc77ca6c59370c0b, State: Initialized, Role: FOLLOWER
I20260620 20:21:54.007239 21027 raft_consensus.cc:348] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.007506 21027 raft_consensus.cc:374] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:54.007575 21027 raft_consensus.cc:729] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6a5d959ff04da3b80b587a1ebf4a7e, State: Initialized, Role: FOLLOWER
I20260620 20:21:54.007548 21025 consensus_queue.cc:260] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.007462 21026 raft_consensus.cc:348] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.007676 21026 raft_consensus.cc:374] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:54.007735 21026 raft_consensus.cc:729] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23f9997046f41d6b3ba2c673db7c398, State: Initialized, Role: FOLLOWER
I20260620 20:21:54.008064 21027 consensus_queue.cc:260] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.008184 21026 consensus_queue.cc:260] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.008514 21025 ts_tablet_manager.cc:1428] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 20:21:54.008806 21008 heartbeater.cc:503] Master 127.2.235.254:42225 was elected leader, sending a full tablet report...
I20260620 20:21:54.008839 20875 heartbeater.cc:503] Master 127.2.235.254:42225 was elected leader, sending a full tablet report...
I20260620 20:21:54.009115 21027 ts_tablet_manager.cc:1428] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 20:21:54.009465 20743 heartbeater.cc:503] Master 127.2.235.254:42225 was elected leader, sending a full tablet report...
I20260620 20:21:54.009883 21026 ts_tablet_manager.cc:1428] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 20:21:54.023401 20876 tablet.cc:2369] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:21:54.103960 20744 tablet.cc:2369] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:54.190119 21032 raft_consensus.cc:482] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:54.190336 21032 raft_consensus.cc:504] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.191372 21032 leader_election.cc:288] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a23f9997046f41d6b3ba2c673db7c398 (127.2.235.194:39711), ad6a5d959ff04da3b80b587a1ebf4a7e (127.2.235.193:41221)
W20260620 20:21:54.192360 21009 tablet.cc:2369] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:54.195011 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:54.191642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:48361 (local address 127.2.235.194:39711)
0620 20:21:54.191747 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:54.191751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:54.191764 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:21:54.191863 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:54.191866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:54.191906 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:54.191988 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:54.192285 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.192886 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.193920 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.194405 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.194496 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:54.194534 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:54.194809 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 20:21:54.194882 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:21:54.195019 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:54.191536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:39711 (local address 127.2.235.195:48361)
0620 20:21:54.191649 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:54.191664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:54.191741 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:54.192000 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:54.192003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:54.192018 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:54.192143 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:54.192150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.192985 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.192988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:54.193805 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:54.193814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.194541 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.194545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:54.194657 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:54.194688 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:21:54.194787 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:21:54.194869 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 20:21:54.195356 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:54.191629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:54921 (local address 127.2.235.193:41221)
0620 20:21:54.191735 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:54.191738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:54.192150 (+ 412us) server_negotiation.cc:407] Connection header received
0620 20:21:54.192218 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:54.192220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:54.192272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:54.192346 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:54.192629 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.193138 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.194249 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.194815 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.194931 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:54.194981 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:54.195199 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:21:54.195275 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 20:21:54.195684 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:54.191545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:41221 (local address 127.2.235.195:54921)
0620 20:21:54.192069 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:54.192079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:54.192151 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:54.192356 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:54.192359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:54.192372 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:54.192522 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:54.192528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.193233 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.193235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:54.194043 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:54.194052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.194942 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.194946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:54.195040 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:54.195069 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:21:54.195507 (+ 438us) client_negotiation.cc:240] Negotiation successful
0620 20:21:54.195576 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":232,"thread_start_us":125,"threads_started":1}
I20260620 20:21:54.195900 20829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2148c1a874bfa9dbc71e5b67881a9" candidate_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23f9997046f41d6b3ba2c673db7c398" is_pre_election: true
I20260620 20:21:54.196197 20829 raft_consensus.cc:2454] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7b6ef0be04e44bc77ca6c59370c0b in term 0.
I20260620 20:21:54.196235 20697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2148c1a874bfa9dbc71e5b67881a9" candidate_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" is_pre_election: true
I20260620 20:21:54.196470 20697 raft_consensus.cc:2454] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b7b6ef0be04e44bc77ca6c59370c0b in term 0.
I20260620 20:21:54.196684 20896 leader_election.cc:302] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23f9997046f41d6b3ba2c673db7c398, d0b7b6ef0be04e44bc77ca6c59370c0b; no voters:
I20260620 20:21:54.196923 21032 raft_consensus.cc:2781] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:54.197001 21032 raft_consensus.cc:482] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:54.197045 21032 raft_consensus.cc:3037] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:54.198060 21032 raft_consensus.cc:504] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.198402 21032 leader_election.cc:288] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [CANDIDATE]: Term 1 election: Requested vote from peers a23f9997046f41d6b3ba2c673db7c398 (127.2.235.194:39711), ad6a5d959ff04da3b80b587a1ebf4a7e (127.2.235.193:41221)
I20260620 20:21:54.198665 20829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2148c1a874bfa9dbc71e5b67881a9" candidate_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a23f9997046f41d6b3ba2c673db7c398"
I20260620 20:21:54.198709 20697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00b2148c1a874bfa9dbc71e5b67881a9" candidate_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e"
I20260620 20:21:54.198781 20829 raft_consensus.cc:3037] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:54.198808 20697 raft_consensus.cc:3037] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:54.200006 20829 raft_consensus.cc:2454] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7b6ef0be04e44bc77ca6c59370c0b in term 1.
I20260620 20:21:54.200006 20697 raft_consensus.cc:2454] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b7b6ef0be04e44bc77ca6c59370c0b in term 1.
I20260620 20:21:54.200307 20896 leader_election.cc:302] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a23f9997046f41d6b3ba2c673db7c398, d0b7b6ef0be04e44bc77ca6c59370c0b; no voters:
I20260620 20:21:54.200460 21032 raft_consensus.cc:2781] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:54.200662 21032 raft_consensus.cc:686] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 1 LEADER]: Becoming Leader. State: Replica: d0b7b6ef0be04e44bc77ca6c59370c0b, State: Running, Role: LEADER
I20260620 20:21:54.200965 21032 consensus_queue.cc:237] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } }
I20260620 20:21:54.204365 20553 catalog_manager.cc:5495] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b7b6ef0be04e44bc77ca6c59370c0b (127.2.235.195). New cstate: current_term: 1 leader_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b7b6ef0be04e44bc77ca6c59370c0b" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 } health_report { overall_health: UNKNOWN } } }
I20260620 20:21:54.217760 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:54.214393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.2.235.195:40855)
0620 20:21:54.214516 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:54.214522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:54.214562 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:21:54.214764 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:54.214766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:54.214810 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:54.214915 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:54.215152 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.215630 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.216419 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:54.216614 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:54.216713 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:54.216992 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:54.217382 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:54.217492 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:54.217547 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 20:21:54.217587 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:21:54.222534 20697 raft_consensus.cc:1261] T 00b2148c1a874bfa9dbc71e5b67881a9 P ad6a5d959ff04da3b80b587a1ebf4a7e [term 1 FOLLOWER]: Refusing update from remote peer d0b7b6ef0be04e44bc77ca6c59370c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:54.222534 20829 raft_consensus.cc:1261] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 1 FOLLOWER]: Refusing update from remote peer d0b7b6ef0be04e44bc77ca6c59370c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:54.223217 21036 consensus_queue.cc:1035] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a23f9997046f41d6b3ba2c673db7c398" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:54.223415 21032 consensus_queue.cc:1035] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6a5d959ff04da3b80b587a1ebf4a7e" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 41221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:54.229893 21041 mvcc.cc:205] Tried to move back new op lower bound from 7299018400649277440 to 7299018400569724928. Current Snapshot: MvccSnapshot[applied={T|T < 7299018400649277440}]
I20260620 20:21:54.237602 20553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:54.237738 20553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:54.239903 20553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b7f9aa59f7f4fd0a62ad66294a453b8: Sending DeleteTablet for 3 replicas of tablet 00b2148c1a874bfa9dbc71e5b67881a9
I20260620 20:21:54.240569 20809 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b2148c1a874bfa9dbc71e5b67881a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:54 UTC) from {username='slave'} at 127.0.0.1:41198
I20260620 20:21:54.240629 20676 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b2148c1a874bfa9dbc71e5b67881a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:54 UTC) from {username='slave'} at 127.0.0.1:39082
I20260620 20:21:54.240986 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20614
I20260620 20:21:54.240995 21049 tablet_replica.cc:330] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: stopping tablet replica
I20260620 20:21:54.241794 21049 raft_consensus.cc:2229] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:54.242091 21049 raft_consensus.cc:2258] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:54.242522 20942 tablet_service.cc:1514] Processing DeleteTablet for tablet 00b2148c1a874bfa9dbc71e5b67881a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:54 UTC) from {username='slave'} at 127.0.0.1:35616
I20260620 20:21:54.244138 21049 ts_tablet_manager.cc:1905] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:54.247469 21049 ts_tablet_manager.cc:1918] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:54.247560 21049 log.cc:1196] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/00b2148c1a874bfa9dbc71e5b67881a9
I20260620 20:21:54.247793 21051 tablet_replica.cc:330] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: stopping tablet replica
I20260620 20:21:54.247840 21049 ts_tablet_manager.cc:1939] T 00b2148c1a874bfa9dbc71e5b67881a9 P a23f9997046f41d6b3ba2c673db7c398: Deleting consensus metadata
I20260620 20:21:54.248119 21051 raft_consensus.cc:2229] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:21:54.248453 21051 raft_consensus.cc:2258] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 20:21:54.248751 20538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:41221 (error 108)
I20260620 20:21:54.248983 20541 catalog_manager.cc:4841] TS a23f9997046f41d6b3ba2c673db7c398 (127.2.235.194:39711): tablet 00b2148c1a874bfa9dbc71e5b67881a9 (table test-table [id=ae7bbe194e27438188726afe7ce4a77d]) successfully deleted
I20260620 20:21:54.249488 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20746
I20260620 20:21:54.250169 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:54.249584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:41221 (local address 127.0.0.1:39084)
0620 20:21:54.250015 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:54.250094 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:41221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329}
I20260620 20:21:54.250571 21051 ts_tablet_manager.cc:1905] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:21:54.250581 20538 catalog_manager.cc:4584] TS ad6a5d959ff04da3b80b587a1ebf4a7e (127.2.235.193:41221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00b2148c1a874bfa9dbc71e5b67881a9: Network error: Client connection negotiation failed: client connection to 127.2.235.193:41221: connect: Connection refused (error 111)
I20260620 20:21:54.253046 21051 ts_tablet_manager.cc:1918] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:54.253141 21051 log.cc:1196] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/00b2148c1a874bfa9dbc71e5b67881a9
I20260620 20:21:54.253507 21051 ts_tablet_manager.cc:1939] T 00b2148c1a874bfa9dbc71e5b67881a9 P d0b7b6ef0be04e44bc77ca6c59370c0b: Deleting consensus metadata
I20260620 20:21:54.254578 20540 catalog_manager.cc:4841] TS d0b7b6ef0be04e44bc77ca6c59370c0b (127.2.235.195:40855): tablet 00b2148c1a874bfa9dbc71e5b67881a9 (table test-table [id=ae7bbe194e27438188726afe7ce4a77d]) successfully deleted
I20260620 20:21:54.255851 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20878
I20260620 20:21:54.260926 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 20523
2026-06-20T20:21:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:54 dist-test-slave-8z53 krb5kdc[21055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:54 dist-test-slave-8z53 krb5kdc[21055](info): set up 2 sockets
Jun 20 20:21:54 dist-test-slave-8z53 krb5kdc[21055](info): commencing operation
krb5kdc: starting...
W20260620 20:21:55.261154 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:21:55.370142 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:46291
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:55.472409 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:55.472640 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:55.472702 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:55.476361 21068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:55.476434 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:55.476468 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:55.476518 21068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:55.476554 21068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:55.481438 21068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:55.482622 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:55.483851 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:55.491745 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:55.491770 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:55.491808 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:55.492063 21068 server_base.cc:1034] running on GCE node
I20260620 20:21:55.492604 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:55.493645 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:55.494848 21068 hybrid_clock.cc:648] HybridClock initialized: now 1781986915494842 us; error 28 us; skew 500 ppm
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:55.497462 21068 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:21:55.498250 21068 webserver.cc:460] Webserver started at http://127.2.235.254:34463/ using document root <none> and password file <none>
I20260620 20:21:55.498730 21068 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:55.498811 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:55.499011 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:55.500838 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "b4dbdf7427e74035beac7fbf4003e558"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "a18a1dab5ae2f91cfd1eee621a8bf531"
server_key_iv: "bf45af3faa6cef99277a8a2b3ed44c3f"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.501387 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "b4dbdf7427e74035beac7fbf4003e558"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "a18a1dab5ae2f91cfd1eee621a8bf531"
server_key_iv: "bf45af3faa6cef99277a8a2b3ed44c3f"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.504487 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:55.506716 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.507869 21068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:55.507998 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "b4dbdf7427e74035beac7fbf4003e558"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "a18a1dab5ae2f91cfd1eee621a8bf531"
server_key_iv: "bf45af3faa6cef99277a8a2b3ed44c3f"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.508100 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:55.532682 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:55.535620 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:55.535805 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:55.542609 21068 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:46291
I20260620 20:21:55.542636 21135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:46291 every 8 connection(s)
I20260620 20:21:55.543445 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:55.545109 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21068
I20260620 20:21:55.545235 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:21:55.545526 2991 external_mini_cluster.cc:1428] Setting key 8ba0378170c8d336d734c44830a1df1b
I20260620 20:21:55.546150 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:55.552248 21136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: Bootstrap starting.
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:55.554387 21136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:55.555176 21136 log.cc:824] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:55.556927 21136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: No bootstrap required, opened a new log
I20260620 20:21:55.558895 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:55.546806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.2.235.254:46291)
0620 20:21:55.547172 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:55.547188 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:55.547225 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:21:55.547949 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:55.547969 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:55.548323 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:55.548667 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:55.549274 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.550734 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.551457 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.552066 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.552211 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:55.554567 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:55.554572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:55.554583 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:55.554607 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:55.556377 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:55.556821 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:55.556826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:55.556830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:55.556928 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:55.557225 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:55.557229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:55.557231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:55.557631 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:55.557843 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:55.558062 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:21:55.558317 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260620 20:21:55.560281 21136 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4dbdf7427e74035beac7fbf4003e558" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46291 } }
I20260620 20:21:55.560503 21136 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:55.560577 21136 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4dbdf7427e74035beac7fbf4003e558, State: Initialized, Role: FOLLOWER
I20260620 20:21:55.560969 21136 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4dbdf7427e74035beac7fbf4003e558" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46291 } }
I20260620 20:21:55.561082 21136 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:55.561167 21136 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:55.561290 21136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:55.562033 21136 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4dbdf7427e74035beac7fbf4003e558" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46291 } }
I20260620 20:21:55.562337 21136 leader_election.cc:302] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4dbdf7427e74035beac7fbf4003e558; no voters:
I20260620 20:21:55.562616 21136 leader_election.cc:288] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:55.562780 21141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:55.563067 21141 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [term 1 LEADER]: Becoming Leader. State: Replica: b4dbdf7427e74035beac7fbf4003e558, State: Running, Role: LEADER
I20260620 20:21:55.563349 21141 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4dbdf7427e74035beac7fbf4003e558" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46291 } }
I20260620 20:21:55.563699 21136 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:55.564888 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4dbdf7427e74035beac7fbf4003e558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4dbdf7427e74035beac7fbf4003e558" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46291 } } }
I20260620 20:21:55.565022 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:55.565423 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4dbdf7427e74035beac7fbf4003e558. Latest consensus state: current_term: 1 leader_uuid: "b4dbdf7427e74035beac7fbf4003e558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4dbdf7427e74035beac7fbf4003e558" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46291 } } }
I20260620 20:21:55.565532 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558 [sys.catalog]: This master's current role is: LEADER
W20260620 20:21:55.568204 21156 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:21:55.568264 21156 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:21:55.568338 21150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:55.568986 21150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:55.574081 21150 catalog_manager.cc:1347] Generated new cluster ID: 6f76021b892144ac85ed12161ead47af
I20260620 20:21:55.574136 21150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:55.585016 21150 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:55.586259 21150 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:55.598481 21150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: Generated new TSK 0
I20260620 20:21:55.599251 21150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:21:55.628150 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46291
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:55.725574 21162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:55.725797 21162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:55.725865 21162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:55.729542 21162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:55.729609 21162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:55.729693 21162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:55.734328 21162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:55.735167 21162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:55.736063 21162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:55.744282 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:55.744354 21162 server_base.cc:1034] running on GCE node
W20260620 20:21:55.744297 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:55.744325 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:55.744863 21162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:55.745455 21162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:55.746603 21162 hybrid_clock.cc:648] HybridClock initialized: now 1781986915746585 us; error 29 us; skew 500 ppm
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:55.748991 21162 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:21:55.749755 21162 webserver.cc:460] Webserver started at http://127.2.235.193:44219/ using document root <none> and password file <none>
I20260620 20:21:55.750218 21162 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:55.750269 21162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:55.750473 21162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:55.752107 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "4072d16487b64dc29b49a194ab3c9d88"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "b6fb784bcf4bf56ca4cd14b6ce2345c7"
server_key_iv: "1fe59898f8f0f6dd0a38579d4b40fc2a"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.752588 21162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "4072d16487b64dc29b49a194ab3c9d88"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "b6fb784bcf4bf56ca4cd14b6ce2345c7"
server_key_iv: "1fe59898f8f0f6dd0a38579d4b40fc2a"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.755836 21162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:21:55.757989 21177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.759061 21162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:21:55.759181 21162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "4072d16487b64dc29b49a194ab3c9d88"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "b6fb784bcf4bf56ca4cd14b6ce2345c7"
server_key_iv: "1fe59898f8f0f6dd0a38579d4b40fc2a"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.759276 21162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:55.769464 21162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:55.771821 21162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:55.771988 21162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:55.772473 21162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:55.773217 21162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:55.773308 21162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.773398 21162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:55.773452 21162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.781524 21162 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:39729
I20260620 20:21:55.781582 21291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:39729 every 8 connection(s)
I20260620 20:21:55.782248 21162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:55.782960 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21162
I20260620 20:21:55.783080 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:21:55.783341 2991 external_mini_cluster.cc:1428] Setting key 9cd15261e561df468ee73e9ce4096fed
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:21:55.796367 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:55.783771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:52751 (local address 127.2.235.254:46291)
0620 20:21:55.783899 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:55.783903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:55.784744 (+ 841us) server_negotiation.cc:407] Connection header received
0620 20:21:55.786152 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:55.786159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:55.786239 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:55.786331 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:55.787398 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.787911 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.789649 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.789895 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.789972 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:55.792586 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:55.792591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:55.792592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:55.792615 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:55.794137 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:55.794703 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:55.794713 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:55.794725 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:55.794816 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:55.795294 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:55.795299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:55.795301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:55.795495 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:55.795605 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:55.796072 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 20:21:55.796194 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:21:55.797039 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:55.784090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:46291 (local address 127.2.235.193:52751)
0620 20:21:55.784601 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:55.784653 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:55.785957 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:55.786465 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:55.786470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:55.786755 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:55.787261 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:55.787277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.788035 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.788053 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:55.789536 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:55.789549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.790041 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.790047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:55.790253 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:55.790833 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:55.790853 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:55.792419 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:55.794271 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:55.794278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:55.794296 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:55.794573 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:55.795027 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:55.795033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:55.795036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:55.795187 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:55.795603 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:55.795610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:55.795844 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 20:21:55.796033 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:21:55.796308 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":112,"threads_started":1}
I20260620 20:21:55.798231 21292 heartbeater.cc:347] Connected to a master server at 127.2.235.254:46291
I20260620 20:21:55.798455 21292 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:55.798856 21292 heartbeater.cc:511] Master 127.2.235.254:46291 requested a full tablet report, sending...
I20260620 20:21:55.800472 21100 ts_manager.cc:194] Registered new tserver with Master: 4072d16487b64dc29b49a194ab3c9d88 (127.2.235.193:39729)
I20260620 20:21:55.802103 21100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:52751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:21:55.809852 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46291
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:55.907662 21297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:55.907866 21297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:55.907902 21297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:55.911485 21297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:55.911530 21297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:55.911593 21297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:55.916286 21297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:55.916978 21297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:55.917709 21297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:55.925390 21302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:55.925369 21303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:55.925374 21305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:55.925617 21297 server_base.cc:1034] running on GCE node
I20260620 20:21:55.926057 21297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:55.926573 21297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:55.927755 21297 hybrid_clock.cc:648] HybridClock initialized: now 1781986915927737 us; error 27 us; skew 500 ppm
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:55.930145 21297 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:21:55.930881 21297 webserver.cc:460] Webserver started at http://127.2.235.194:36405/ using document root <none> and password file <none>
I20260620 20:21:55.931335 21297 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:55.931413 21297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:55.931604 21297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:55.933236 21297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "aff71d1bb5621a24a62ed06b3c20f7b8"
server_key_iv: "d700fe2c466972b7afb8074899a19410"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.933782 21297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "aff71d1bb5621a24a62ed06b3c20f7b8"
server_key_iv: "d700fe2c466972b7afb8074899a19410"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.936951 21297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:21:55.939010 21312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.940076 21297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:55.940204 21297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912"
format_stamp: "Formatted at 2026-06-20 20:21:55 on dist-test-slave-8z53"
server_key: "aff71d1bb5621a24a62ed06b3c20f7b8"
server_key_iv: "d700fe2c466972b7afb8074899a19410"
server_key_version: "encryptionkey@0"
I20260620 20:21:55.940300 21297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:55.950956 21297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:55.953148 21297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:55.953351 21297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:55.953897 21297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:55.954698 21297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:55.954769 21297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.954834 21297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:55.954865 21297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:55.962913 21297 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:34247
I20260620 20:21:55.962939 21426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:34247 every 8 connection(s)
I20260620 20:21:55.963732 21297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:55.964795 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21297
I20260620 20:21:55.964931 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:21:55.965245 2991 external_mini_cluster.cc:1428] Setting key 85dd37319f48300e8c04fa41160add92
Jun 20 20:21:55 dist-test-slave-8z53 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986915, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:21:55.976958 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:55.965411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:57437 (local address 127.2.235.254:46291)
0620 20:21:55.965562 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:55.965566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:55.966320 (+ 754us) server_negotiation.cc:407] Connection header received
0620 20:21:55.967505 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:55.967508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:55.967558 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:55.967622 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:55.968997 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.969536 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.971140 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.971325 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.971391 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:55.973703 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:55.973706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:55.973708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:55.973737 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:55.975140 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:55.975624 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:55.975626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:55.975627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:55.975692 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:55.976004 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:55.976007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:55.976008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:55.976175 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:55.976244 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:55.976689 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 20:21:55.976778 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:21:55.977669 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:55.965794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:46291 (local address 127.2.235.194:57437)
0620 20:21:55.966203 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:55.966246 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:55.967316 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:55.967770 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:55.967780 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:55.968237 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:55.968852 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:55.968873 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.969675 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.969682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:55.971031 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:55.971044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:55.971460 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:55.971466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:55.971667 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:55.972148 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:55.972167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:55.973550 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:55.975261 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:55.975265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:55.975275 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:55.975533 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:55.975763 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:55.975765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:55.975766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:55.975878 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:55.976253 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:55.976257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:55.976471 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 20:21:55.976655 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 20:21:55.976916 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260620 20:21:55.978710 21427 heartbeater.cc:347] Connected to a master server at 127.2.235.254:46291
I20260620 20:21:55.978909 21427 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:55.979343 21427 heartbeater.cc:511] Master 127.2.235.254:46291 requested a full tablet report, sending...
I20260620 20:21:55.980285 21100 ts_manager.cc:194] Registered new tserver with Master: f1d3ac3a9daf4a57aa4b77f2fae1d912 (127.2.235.194:34247)
I20260620 20:21:55.980845 21100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:57437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:21:55.990046 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46291
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:56.089215 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:56.089479 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:56.089550 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:56.093339 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:56.093405 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:56.093489 21432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:56.098434 21432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:56.099249 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:56.100185 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:56.108760 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.108780 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.108780 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:56.108899 21432 server_base.cc:1034] running on GCE node
I20260620 20:21:56.109359 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:56.109913 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:56.111091 21432 hybrid_clock.cc:648] HybridClock initialized: now 1781986916111071 us; error 34 us; skew 500 ppm
Jun 20 20:21:56 dist-test-slave-8z53 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:56.117414 21432 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:21:56.118173 21432 webserver.cc:460] Webserver started at http://127.2.235.195:34913/ using document root <none> and password file <none>
I20260620 20:21:56.118682 21432 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:56.118772 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:56.118966 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:56.120687 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "cb0091ef38524d38afb6a4f4d3764e61"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "6a5a83f0c6458b750a086998ef689097"
server_key_iv: "70ed2619f510e50c9a5ef49d6776be24"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.121207 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "cb0091ef38524d38afb6a4f4d3764e61"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "6a5a83f0c6458b750a086998ef689097"
server_key_iv: "70ed2619f510e50c9a5ef49d6776be24"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.124395 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:21:56.126605 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.127686 21432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:21:56.127841 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "cb0091ef38524d38afb6a4f4d3764e61"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "6a5a83f0c6458b750a086998ef689097"
server_key_iv: "70ed2619f510e50c9a5ef49d6776be24"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.127939 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:56.139870 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:56.142320 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:56.142481 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:56.142953 21432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:56.143708 21432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:56.143774 21432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.143838 21432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:56.143888 21432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.151894 21432 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:42041
I20260620 20:21:56.151921 21561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:42041 every 8 connection(s)
I20260620 20:21:56.152637 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:21:56.154783 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21432
I20260620 20:21:56.154906 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:21:56.155189 2991 external_mini_cluster.cc:1428] Setting key 4070a9daec6fa15f202243b2c542babd
Jun 20 20:21:56 dist-test-slave-8z53 krb5kdc[21055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:56.165117 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.154177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:58985 (local address 127.2.235.254:46291)
0620 20:21:56.154325 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.154330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.155182 (+ 852us) server_negotiation.cc:407] Connection header received
0620 20:21:56.156455 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.156458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.156514 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.156625 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:56.157825 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.158330 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.159386 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.159606 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.159705 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.161779 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:56.161782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:56.161785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:56.161815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:56.163226 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:56.163691 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:56.163696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:56.163698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:56.163773 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:56.164096 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:56.164101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:56.164102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:56.164266 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:56.164402 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.164849 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.164961 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:21:56.165836 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.154499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:46291 (local address 127.2.235.195:58985)
0620 20:21:56.155047 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.155085 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.156233 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.156783 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.156790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.157206 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:56.157690 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.157701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.158446 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.158450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.159246 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.159254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.159735 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.159738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.159889 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.160358 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:56.160372 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:56.161636 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:56.163331 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:56.163339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:56.163350 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:56.163582 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:56.163864 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:56.163868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:56.163870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:56.164003 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:56.164401 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:56.164407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:56.164626 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.164794 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.165075 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":148,"threads_started":1}
I20260620 20:21:56.166739 21562 heartbeater.cc:347] Connected to a master server at 127.2.235.254:46291
I20260620 20:21:56.166951 21562 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:56.167326 21562 heartbeater.cc:511] Master 127.2.235.254:46291 requested a full tablet report, sending...
I20260620 20:21:56.168229 21100 ts_manager.cc:194] Registered new tserver with Master: cb0091ef38524d38afb6a4f4d3764e61 (127.2.235.195:42041)
I20260620 20:21:56.168735 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:21:56.168915 21100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:58985
I20260620 20:21:56.176930 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.170783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.2.235.254:46291)
0620 20:21:56.170882 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.170886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.171024 (+ 138us) server_negotiation.cc:407] Connection header received
0620 20:21:56.171105 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.171107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.171134 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.171190 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:56.171723 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.172138 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.172836 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.173009 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.173072 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.174288 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:56.174290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:56.174292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:56.174310 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:56.175711 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:56.176088 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:56.176090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:56.176091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:56.176151 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:56.176379 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:56.176381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:56.176382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:56.176522 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:56.176623 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.176777 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.176845 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:21:56.179322 21100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:21:56.182128 21100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:21:56.194124 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.188945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:42041 (local address 127.0.0.1:56280)
0620 20:21:56.189234 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.189293 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.189404 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.190363 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.190368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.190390 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:56.190505 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.190514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.191635 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.191639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.192593 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.192602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.193682 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.193686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.193780 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.193812 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.193918 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.194012 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260620 20:21:56.194150 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.189013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.2.235.195:42041)
0620 20:21:56.189766 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.189780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.189801 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:21:56.189875 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.189880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.190202 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.190637 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:56.190753 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.191514 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.192718 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.193547 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.193678 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.193807 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.193956 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.194035 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":99,"threads_started":1}
I20260620 20:21:56.195030 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.189371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:34247 (local address 127.0.0.1:54196)
0620 20:21:56.189653 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.189666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.189756 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.190816 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.190819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.190847 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:56.190990 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.190998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.192334 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.192337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.193480 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.193491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.194650 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.194653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.194734 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.194754 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.194841 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.194908 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":51,"spinlock_wait_cycles":5120,"thread_start_us":73,"threads_started":1}
I20260620 20:21:56.195084 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.189785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.2.235.194:34247)
0620 20:21:56.190173 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.190183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.190200 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:21:56.190282 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.190287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.190667 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.190910 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:56.191091 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.192190 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.193602 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.194527 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.194649 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.194769 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.194865 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.194952 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
I20260620 20:21:56.195369 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.189445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39729 (local address 127.0.0.1:35724)
0620 20:21:56.190015 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.190027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.190119 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.190820 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.190822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.190848 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:56.190964 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.190970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.192359 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.192362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.193547 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.193556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.194982 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.194985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.195080 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.195101 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.195170 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.195237 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":27,"spinlock_wait_cycles":6400,"thread_start_us":93,"threads_started":1}
I20260620 20:21:56.195771 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.189665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.2.235.193:39729)
0620 20:21:56.190013 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.190024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.190096 (+ 72us) server_negotiation.cc:407] Connection header received
0620 20:21:56.190274 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.190282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.190464 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.190637 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:56.191082 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.192190 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.193668 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.194707 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.194805 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.194921 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.195547 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.195645 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":80,"threads_started":1}
I20260620 20:21:56.196940 21495 tablet_service.cc:1467] Processing CreateTablet for tablet e38af5922c744a25bd783674c2175a24 (DEFAULT_TABLE table=test-table [id=7be7093bc948475bb72df703ea616aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:56.197933 21495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38af5922c744a25bd783674c2175a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:56.197950 21360 tablet_service.cc:1467] Processing CreateTablet for tablet e38af5922c744a25bd783674c2175a24 (DEFAULT_TABLE table=test-table [id=7be7093bc948475bb72df703ea616aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:56.198171 21225 tablet_service.cc:1467] Processing CreateTablet for tablet e38af5922c744a25bd783674c2175a24 (DEFAULT_TABLE table=test-table [id=7be7093bc948475bb72df703ea616aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:56.198637 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38af5922c744a25bd783674c2175a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:56.199007 21225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38af5922c744a25bd783674c2175a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:56.203687 21579 tablet_bootstrap.cc:492] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Bootstrap starting.
I20260620 20:21:56.205027 21580 tablet_bootstrap.cc:492] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Bootstrap starting.
I20260620 20:21:56.205571 21581 tablet_bootstrap.cc:492] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: Bootstrap starting.
I20260620 20:21:56.206288 21579 tablet_bootstrap.cc:654] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:56.206849 21580 tablet_bootstrap.cc:654] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:56.207146 21579 log.cc:824] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:56.208047 21580 log.cc:824] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:56.208118 21581 tablet_bootstrap.cc:654] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:56.208894 21581 log.cc:824] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:56.209161 21579 tablet_bootstrap.cc:492] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: No bootstrap required, opened a new log
I20260620 20:21:56.209380 21579 ts_tablet_manager.cc:1397] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:21:56.209738 21580 tablet_bootstrap.cc:492] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: No bootstrap required, opened a new log
I20260620 20:21:56.209869 21580 ts_tablet_manager.cc:1397] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:56.210775 21581 tablet_bootstrap.cc:492] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: No bootstrap required, opened a new log
I20260620 20:21:56.211012 21581 ts_tablet_manager.cc:1397] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:21:56.212778 21580 raft_consensus.cc:348] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.212955 21580 raft_consensus.cc:374] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:56.212988 21580 raft_consensus.cc:729] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0091ef38524d38afb6a4f4d3764e61, State: Initialized, Role: FOLLOWER
I20260620 20:21:56.213008 21579 raft_consensus.cc:348] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.213279 21579 raft_consensus.cc:374] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:56.213342 21579 raft_consensus.cc:729] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d3ac3a9daf4a57aa4b77f2fae1d912, State: Initialized, Role: FOLLOWER
I20260620 20:21:56.213433 21580 consensus_queue.cc:260] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.213739 21579 consensus_queue.cc:260] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.214075 21562 heartbeater.cc:503] Master 127.2.235.254:46291 was elected leader, sending a full tablet report...
I20260620 20:21:56.214350 21427 heartbeater.cc:503] Master 127.2.235.254:46291 was elected leader, sending a full tablet report...
I20260620 20:21:56.214365 21580 ts_tablet_manager.cc:1428] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 20:21:56.214644 21579 ts_tablet_manager.cc:1428] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 20:21:56.214841 21428 tablet.cc:2369] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:56.215283 21581 raft_consensus.cc:348] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.215545 21581 raft_consensus.cc:374] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:56.215608 21581 raft_consensus.cc:729] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4072d16487b64dc29b49a194ab3c9d88, State: Initialized, Role: FOLLOWER
I20260620 20:21:56.215981 21581 consensus_queue.cc:260] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.216591 21581 ts_tablet_manager.cc:1428] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:56.216993 21292 heartbeater.cc:503] Master 127.2.235.254:46291 was elected leader, sending a full tablet report...
W20260620 20:21:56.283175 21293 tablet.cc:2369] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:56.301715 21585 raft_consensus.cc:482] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:56.301864 21585 raft_consensus.cc:504] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.302867 21585 leader_election.cc:288] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4072d16487b64dc29b49a194ab3c9d88 (127.2.235.193:39729), f1d3ac3a9daf4a57aa4b77f2fae1d912 (127.2.235.194:34247)
I20260620 20:21:56.306702 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.303088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:34247 (local address 127.2.235.195:46781)
0620 20:21:56.303511 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.303523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.303619 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.303854 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.303856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.303868 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:56.303970 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.303974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.304748 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.304750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.305635 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.305644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.306314 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.306317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.306417 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.306440 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.306516 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.306575 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":67,"thread_start_us":125,"threads_started":1}
I20260620 20:21:56.306721 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.303170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:46781 (local address 127.2.235.194:34247)
0620 20:21:56.303283 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.303287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.303610 (+ 323us) server_negotiation.cc:407] Connection header received
0620 20:21:56.303702 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.303705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.303757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.303839 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:56.304102 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.304611 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.305748 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.306209 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.306297 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.306334 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.306542 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.306605 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:21:56.306973 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.303159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:59211 (local address 127.2.235.193:39729)
0620 20:21:56.303283 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.303287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.303384 (+ 97us) server_negotiation.cc:407] Connection header received
0620 20:21:56.303513 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.303517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.303577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.303652 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:56.304132 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.304603 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.306040 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.306449 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.306546 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.306585 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.306789 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.306880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:21:56.306955 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.303088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39729 (local address 127.2.235.195:59211)
0620 20:21:56.303296 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.303314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.303418 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.303675 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.303679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.303697 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:56.303916 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.303926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.304713 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.304716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.305928 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.305938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.306553 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.306556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.306660 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.306679 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.306761 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.306835 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":70}
I20260620 20:21:56.307632 21245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38af5922c744a25bd783674c2175a24" candidate_uuid: "cb0091ef38524d38afb6a4f4d3764e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072d16487b64dc29b49a194ab3c9d88" is_pre_election: true
I20260620 20:21:56.307657 21381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38af5922c744a25bd783674c2175a24" candidate_uuid: "cb0091ef38524d38afb6a4f4d3764e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" is_pre_election: true
I20260620 20:21:56.307891 21245 raft_consensus.cc:2454] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0091ef38524d38afb6a4f4d3764e61 in term 0.
I20260620 20:21:56.307890 21381 raft_consensus.cc:2454] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0091ef38524d38afb6a4f4d3764e61 in term 0.
I20260620 20:21:56.308310 21451 leader_election.cc:302] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0091ef38524d38afb6a4f4d3764e61, f1d3ac3a9daf4a57aa4b77f2fae1d912; no voters:
I20260620 20:21:56.308605 21585 raft_consensus.cc:2781] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:56.308684 21585 raft_consensus.cc:482] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:56.308713 21585 raft_consensus.cc:3037] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:56.309702 21585 raft_consensus.cc:504] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.310009 21585 leader_election.cc:288] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [CANDIDATE]: Term 1 election: Requested vote from peers 4072d16487b64dc29b49a194ab3c9d88 (127.2.235.193:39729), f1d3ac3a9daf4a57aa4b77f2fae1d912 (127.2.235.194:34247)
I20260620 20:21:56.310312 21245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38af5922c744a25bd783674c2175a24" candidate_uuid: "cb0091ef38524d38afb6a4f4d3764e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4072d16487b64dc29b49a194ab3c9d88"
I20260620 20:21:56.310381 21381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38af5922c744a25bd783674c2175a24" candidate_uuid: "cb0091ef38524d38afb6a4f4d3764e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912"
I20260620 20:21:56.310431 21245 raft_consensus.cc:3037] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:56.310472 21381 raft_consensus.cc:3037] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:56.311626 21245 raft_consensus.cc:2454] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0091ef38524d38afb6a4f4d3764e61 in term 1.
I20260620 20:21:56.311622 21381 raft_consensus.cc:2454] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0091ef38524d38afb6a4f4d3764e61 in term 1.
I20260620 20:21:56.311900 21451 leader_election.cc:302] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0091ef38524d38afb6a4f4d3764e61, f1d3ac3a9daf4a57aa4b77f2fae1d912; no voters:
I20260620 20:21:56.312072 21585 raft_consensus.cc:2781] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:56.312290 21585 raft_consensus.cc:686] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 1 LEADER]: Becoming Leader. State: Replica: cb0091ef38524d38afb6a4f4d3764e61, State: Running, Role: LEADER
I20260620 20:21:56.312577 21585 consensus_queue.cc:237] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } }
I20260620 20:21:56.315397 21098 catalog_manager.cc:5495] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0091ef38524d38afb6a4f4d3764e61 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "cb0091ef38524d38afb6a4f4d3764e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb0091ef38524d38afb6a4f4d3764e61" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 } health_report { overall_health: UNKNOWN } } }
I20260620 20:21:56.344987 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.341787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.2.235.195:42041)
0620 20:21:56.341941 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.341944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.341960 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:21:56.342096 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.342098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.342141 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.342218 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:56.342421 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.342876 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.343650 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.343849 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.343920 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.344226 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:56.344591 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:56.344696 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.344797 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.344853 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:21:56.350270 21245 raft_consensus.cc:1261] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 1 FOLLOWER]: Refusing update from remote peer cb0091ef38524d38afb6a4f4d3764e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:56.350258 21381 raft_consensus.cc:1261] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 1 FOLLOWER]: Refusing update from remote peer cb0091ef38524d38afb6a4f4d3764e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:56.350963 21585 consensus_queue.cc:1035] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4072d16487b64dc29b49a194ab3c9d88" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:56.351138 21589 consensus_queue.cc:1035] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d3ac3a9daf4a57aa4b77f2fae1d912" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 34247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:56.356735 21592 mvcc.cc:205] Tried to move back new op lower bound from 7299018409363640320 to 7299018409218342912. Current Snapshot: MvccSnapshot[applied={T|T < 7299018409363640320}]
I20260620 20:21:56.365449 21099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:56.365566 21099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:56.367733 21099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4dbdf7427e74035beac7fbf4003e558: Sending DeleteTablet for 3 replicas of tablet e38af5922c744a25bd783674c2175a24
I20260620 20:21:56.368413 21495 tablet_service.cc:1514] Processing DeleteTablet for tablet e38af5922c744a25bd783674c2175a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:56 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:56280
I20260620 20:21:56.368413 21225 tablet_service.cc:1514] Processing DeleteTablet for tablet e38af5922c744a25bd783674c2175a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:56 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:35724
I20260620 20:21:56.368597 21360 tablet_service.cc:1514] Processing DeleteTablet for tablet e38af5922c744a25bd783674c2175a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:56 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:54196
I20260620 20:21:56.369024 21603 tablet_replica.cc:330] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: stopping tablet replica
I20260620 20:21:56.369040 21602 tablet_replica.cc:330] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88: stopping tablet replica
I20260620 20:21:56.369311 21603 raft_consensus.cc:2229] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:56.369321 21602 raft_consensus.cc:2229] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:56.369568 21603 raft_consensus.cc:2258] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:56.369632 21602 raft_consensus.cc:2258] T e38af5922c744a25bd783674c2175a24 P 4072d16487b64dc29b49a194ab3c9d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:56.369756 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21162
I20260620 20:21:56.369990 21601 tablet_replica.cc:330] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: stopping tablet replica
I20260620 20:21:56.370206 21601 raft_consensus.cc:2229] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:21:56.370494 21601 raft_consensus.cc:2258] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:56.371935 21601 ts_tablet_manager.cc:1905] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:56.372727 21603 ts_tablet_manager.cc:1905] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:56.374699 21601 ts_tablet_manager.cc:1918] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:56.374788 21601 log.cc:1196] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/e38af5922c744a25bd783674c2175a24
I20260620 20:21:56.375092 21601 ts_tablet_manager.cc:1939] T e38af5922c744a25bd783674c2175a24 P cb0091ef38524d38afb6a4f4d3764e61: Deleting consensus metadata
I20260620 20:21:56.375849 21603 ts_tablet_manager.cc:1918] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:56.375938 21603 log.cc:1196] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/e38af5922c744a25bd783674c2175a24
I20260620 20:21:56.376091 21085 catalog_manager.cc:4841] TS cb0091ef38524d38afb6a4f4d3764e61 (127.2.235.195:42041): tablet e38af5922c744a25bd783674c2175a24 (table test-table [id=7be7093bc948475bb72df703ea616aae]) successfully deleted
I20260620 20:21:56.376354 21603 ts_tablet_manager.cc:1939] T e38af5922c744a25bd783674c2175a24 P f1d3ac3a9daf4a57aa4b77f2fae1d912: Deleting consensus metadata
I20260620 20:21:56.377404 21087 catalog_manager.cc:4841] TS f1d3ac3a9daf4a57aa4b77f2fae1d912 (127.2.235.194:34247): tablet e38af5922c744a25bd783674c2175a24 (table test-table [id=7be7093bc948475bb72df703ea616aae]) successfully deleted
W20260620 20:21:56.379451 21086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:39729 (error 108)
I20260620 20:21:56.379680 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21297
I20260620 20:21:56.381038 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.380812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:39729 (local address 127.0.0.1:35740)
0620 20:21:56.380914 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.380983 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:39729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:21:56.381268 21086 catalog_manager.cc:4584] TS 4072d16487b64dc29b49a194ab3c9d88 (127.2.235.193:39729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e38af5922c744a25bd783674c2175a24: Network error: Client connection negotiation failed: client connection to 127.2.235.193:39729: connect: Connection refused (error 111)
I20260620 20:21:56.384606 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21432
I20260620 20:21:56.389576 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21068
2026-06-20T20:21:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2129 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T20:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:56Z Disabled control of system clock
I20260620 20:21:56.417002 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:37569
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:21:56.516120 21608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:56.516326 21608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:56.516355 21608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:56.519891 21608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:56.519932 21608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:56.519945 21608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:56.519959 21608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:56.519972 21608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:56.524927 21608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:37569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:56.525851 21608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:56.526984 21608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:56.535909 21614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.535917 21616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:56.536353 21608 server_base.cc:1034] running on GCE node
W20260620 20:21:56.536391 21613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:56.536960 21608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:56.537983 21608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:56.539170 21608 hybrid_clock.cc:648] HybridClock initialized: now 1781986916539144 us; error 45 us; skew 500 ppm
I20260620 20:21:56.540812 21608 webserver.cc:460] Webserver started at http://127.2.235.254:35159/ using document root <none> and password file <none>
I20260620 20:21:56.541360 21608 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:56.541442 21608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:56.541635 21608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:56.543286 21608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "dc1ae3824e054015b4662c89be95edff"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "674b2ebcc40cf5c03e2d52c99507dc4c"
server_key_iv: "de5270fa948c0e2e7c2f739afe6bf9e9"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.543769 21608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "dc1ae3824e054015b4662c89be95edff"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "674b2ebcc40cf5c03e2d52c99507dc4c"
server_key_iv: "de5270fa948c0e2e7c2f739afe6bf9e9"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.546923 21608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:21:56.548861 21622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.549842 21608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:56.549960 21608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "dc1ae3824e054015b4662c89be95edff"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "674b2ebcc40cf5c03e2d52c99507dc4c"
server_key_iv: "de5270fa948c0e2e7c2f739afe6bf9e9"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.550071 21608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:56.572252 21608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:56.572804 21608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:56.572973 21608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:56.579188 21608 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:37569
I20260620 20:21:56.579195 21675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:37569 every 8 connection(s)
I20260620 20:21:56.580031 21608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:56.582883 21676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:56.585047 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21608
I20260620 20:21:56.585161 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:21:56.585500 2991 external_mini_cluster.cc:1428] Setting key 4d610496ee26dfea140778e3bf2df666
I20260620 20:21:56.588577 21676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff: Bootstrap starting.
I20260620 20:21:56.590636 21676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:56.591358 21676 log.cc:824] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:56.593029 21676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff: No bootstrap required, opened a new log
I20260620 20:21:56.594210 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.586854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.2.235.254:37569)
0620 20:21:56.587442 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.587456 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.587488 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:21:56.588084 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.588100 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.588455 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.588784 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:56.589169 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.590142 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.591079 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.591568 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.591744 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.592568 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:56.592573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:56.592594 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:56.592949 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:56.593001 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:56.593020 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:56.593123 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.593304 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.593566 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":121,"threads_started":1}
I20260620 20:21:56.596622 21676 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1ae3824e054015b4662c89be95edff" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37569 } }
I20260620 20:21:56.596855 21676 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:56.596939 21676 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1ae3824e054015b4662c89be95edff, State: Initialized, Role: FOLLOWER
I20260620 20:21:56.597395 21676 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1ae3824e054015b4662c89be95edff" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37569 } }
I20260620 20:21:56.597544 21676 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:56.597625 21676 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:56.597743 21676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:56.598539 21676 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1ae3824e054015b4662c89be95edff" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37569 } }
I20260620 20:21:56.598860 21676 leader_election.cc:302] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1ae3824e054015b4662c89be95edff; no voters:
I20260620 20:21:56.599151 21676 leader_election.cc:288] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:56.599296 21681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:56.599532 21681 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [term 1 LEADER]: Becoming Leader. State: Replica: dc1ae3824e054015b4662c89be95edff, State: Running, Role: LEADER
I20260620 20:21:56.599819 21681 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1ae3824e054015b4662c89be95edff" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37569 } }
I20260620 20:21:56.600296 21676 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:56.601421 21682 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1ae3824e054015b4662c89be95edff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1ae3824e054015b4662c89be95edff" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37569 } } }
I20260620 20:21:56.601526 21682 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:56.601814 21683 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1ae3824e054015b4662c89be95edff. Latest consensus state: current_term: 1 leader_uuid: "dc1ae3824e054015b4662c89be95edff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1ae3824e054015b4662c89be95edff" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 37569 } } }
I20260620 20:21:56.601886 21690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:56.601909 21683 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:56.604928 21690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:56.610198 21690 catalog_manager.cc:1347] Generated new cluster ID: 96824cda96fa4d67b53b3290442b3d4d
I20260620 20:21:56.610258 21690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:56.624190 21690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:56.625267 21690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:56.635756 21690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff: Generated new TSK 0
I20260620 20:21:56.636372 21690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:21:56.643085 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37569
--builtin_ntp_servers=127.2.235.212:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:21:56.744560 21700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:56.744802 21700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:56.744871 21700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:56.748514 21700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:56.748572 21700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:56.748643 21700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:56.752914 21700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:56.753784 21700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:56.754979 21700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:56.763477 21705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.763497 21706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.763484 21708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:56.763722 21700 server_base.cc:1034] running on GCE node
I20260620 20:21:56.764168 21700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:56.764711 21700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:56.765930 21700 hybrid_clock.cc:648] HybridClock initialized: now 1781986916765916 us; error 32 us; skew 500 ppm
I20260620 20:21:56.767609 21700 webserver.cc:460] Webserver started at http://127.2.235.193:43863/ using document root <none> and password file <none>
I20260620 20:21:56.768146 21700 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:56.768229 21700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:56.768432 21700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:56.770239 21700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "915a1f7504bd434197dd7b6338f9d2ac"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "6f6204b318a1e6fac68f970e9bce8a99"
server_key_iv: "71fbe78f76794c7682251a2420a7403c"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.770740 21700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "915a1f7504bd434197dd7b6338f9d2ac"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "6f6204b318a1e6fac68f970e9bce8a99"
server_key_iv: "71fbe78f76794c7682251a2420a7403c"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.773944 21700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:56.776082 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.777104 21700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:56.777230 21700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "915a1f7504bd434197dd7b6338f9d2ac"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "6f6204b318a1e6fac68f970e9bce8a99"
server_key_iv: "71fbe78f76794c7682251a2420a7403c"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.777380 21700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:56.807399 21700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:56.807967 21700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:56.808138 21700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:56.808635 21700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:56.809523 21700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:56.809592 21700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.809656 21700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:56.809705 21700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.818274 21700 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:38033
I20260620 20:21:56.818311 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:38033 every 8 connection(s)
I20260620 20:21:56.819100 21700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:56.827191 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.820835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50713 (local address 127.2.235.254:37569)
0620 20:21:56.820973 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.820979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.821999 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 20:21:56.823129 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.823132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.823175 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.823249 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:56.823970 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.824468 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.825632 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.825857 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.825946 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.826843 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:56.826845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:56.826847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:56.826857 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:56.826874 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:56.826881 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:56.826933 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.827045 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.827088 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:21:56.827880 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.821209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37569 (local address 127.2.235.193:50713)
0620 20:21:56.821845 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.821878 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.822954 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.823383 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.823389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.823470 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:56.823863 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.823874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.824611 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.824615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.825531 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.825541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.825988 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.825990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.826112 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.826618 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:21:56.826620 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:56.826653 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:21:56.826702 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:56.826944 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:56.826951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:56.826954 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.827032 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.827212 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260620 20:21:56.828928 21828 heartbeater.cc:347] Connected to a master server at 127.2.235.254:37569
I20260620 20:21:56.829134 21828 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:56.829339 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21700
I20260620 20:21:56.829437 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:21:56.829656 21828 heartbeater.cc:511] Master 127.2.235.254:37569 requested a full tablet report, sending...
I20260620 20:21:56.829684 2991 external_mini_cluster.cc:1428] Setting key 45482e99328bccd0eca5bd24b1e4a0b3
I20260620 20:21:56.831260 21638 ts_manager.cc:194] Registered new tserver with Master: 915a1f7504bd434197dd7b6338f9d2ac (127.2.235.193:38033)
I20260620 20:21:56.831607 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37569
--builtin_ntp_servers=127.2.235.212:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:21:56.832736 21638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.193:50713
W20260620 20:21:56.931974 21831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:56.932161 21831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:56.932191 21831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:56.936039 21831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:56.936082 21831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:56.936128 21831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:56.940701 21831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:56.941584 21831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:56.942800 21831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:56.951180 21836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.951216 21839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:56.951156 21837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:56.951602 21831 server_base.cc:1034] running on GCE node
I20260620 20:21:56.951951 21831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:56.952478 21831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:56.953644 21831 hybrid_clock.cc:648] HybridClock initialized: now 1781986916953615 us; error 43 us; skew 500 ppm
I20260620 20:21:56.955322 21831 webserver.cc:460] Webserver started at http://127.2.235.194:32779/ using document root <none> and password file <none>
I20260620 20:21:56.955875 21831 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:56.955962 21831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:56.956172 21831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:56.958012 21831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "b3ef6f38ea904626aa00152aeabc832d"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "2e9b52545262aa7d919353baa086986b"
server_key_iv: "69ac0910b55058d62d56ed07d6d2cb0d"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.958508 21831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "b3ef6f38ea904626aa00152aeabc832d"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "2e9b52545262aa7d919353baa086986b"
server_key_iv: "69ac0910b55058d62d56ed07d6d2cb0d"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.961620 21831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:56.963660 21845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.964694 21831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:56.964843 21831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "b3ef6f38ea904626aa00152aeabc832d"
format_stamp: "Formatted at 2026-06-20 20:21:56 on dist-test-slave-8z53"
server_key: "2e9b52545262aa7d919353baa086986b"
server_key_iv: "69ac0910b55058d62d56ed07d6d2cb0d"
server_key_version: "encryptionkey@0"
I20260620 20:21:56.964948 21831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:56.976354 21831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:56.976886 21831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:56.977058 21831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:56.977626 21831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:56.978394 21831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:56.978456 21831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.978513 21831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:56.978551 21831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:56.986557 21831 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:42885
I20260620 20:21:56.986593 21958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:42885 every 8 connection(s)
I20260620 20:21:56.987275 21831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:56.995275 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.988799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:42517 (local address 127.2.235.254:37569)
0620 20:21:56.988935 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:56.988939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:56.989963 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 20:21:56.991236 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:56.991240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:56.991284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:56.991353 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:56.992138 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.992599 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.993691 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.993894 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.993967 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.994891 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:56.994894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:56.994896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:56.994907 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:56.994925 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:56.994932 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:56.994988 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:56.995121 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 20:21:56.995168 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:21:56.996057 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:56.989132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37569 (local address 127.2.235.194:42517)
0620 20:21:56.989832 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:56.989870 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:56.991062 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:56.991499 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:56.991505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:56.991579 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:56.992030 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.992041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.992701 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.992704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.993586 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:56.993596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:56.994013 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:56.994015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:56.994137 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:56.994660 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:21:56.994662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:56.994697 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:21:56.994758 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:56.995005 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:56.995011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:56.995015 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:21:56.995104 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:21:56.995293 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260620 20:21:56.997131 21959 heartbeater.cc:347] Connected to a master server at 127.2.235.254:37569
I20260620 20:21:56.997299 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21831
I20260620 20:21:56.997364 21959 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:56.997404 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:21:56.997685 2991 external_mini_cluster.cc:1428] Setting key 04b1787e78488057bbb979908aacb241
I20260620 20:21:56.997854 21959 heartbeater.cc:511] Master 127.2.235.254:37569 requested a full tablet report, sending...
I20260620 20:21:56.998814 21639 ts_manager.cc:194] Registered new tserver with Master: b3ef6f38ea904626aa00152aeabc832d (127.2.235.194:42885)
I20260620 20:21:56.999389 21639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.194:42517
I20260620 20:21:56.999624 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37569
--builtin_ntp_servers=127.2.235.212:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:21:57.099716 21962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:57.099938 21962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:57.099969 21962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:57.104087 21962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:57.104138 21962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:57.104188 21962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:57.108673 21962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:57.109555 21962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:57.110785 21962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:57.119371 21970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:57.119467 21962 server_base.cc:1034] running on GCE node
W20260620 20:21:57.119383 21968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:57.119383 21967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:57.119977 21962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:57.120515 21962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:57.121673 21962 hybrid_clock.cc:648] HybridClock initialized: now 1781986917121636 us; error 48 us; skew 500 ppm
I20260620 20:21:57.123358 21962 webserver.cc:460] Webserver started at http://127.2.235.195:38281/ using document root <none> and password file <none>
I20260620 20:21:57.123831 21962 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:57.123885 21962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:57.124032 21962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:57.125928 21962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "c094469112a04befa382d7d4376c442d"
format_stamp: "Formatted at 2026-06-20 20:21:57 on dist-test-slave-8z53"
server_key: "0746cabe730f15215d3ee74816106e3a"
server_key_iv: "b29a2934c5a5eb1bd3bdfc91843831d0"
server_key_version: "encryptionkey@0"
I20260620 20:21:57.126382 21962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "c094469112a04befa382d7d4376c442d"
format_stamp: "Formatted at 2026-06-20 20:21:57 on dist-test-slave-8z53"
server_key: "0746cabe730f15215d3ee74816106e3a"
server_key_iv: "b29a2934c5a5eb1bd3bdfc91843831d0"
server_key_version: "encryptionkey@0"
I20260620 20:21:57.129595 21962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:21:57.131747 21976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:57.132773 21962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:57.132862 21962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "c094469112a04befa382d7d4376c442d"
format_stamp: "Formatted at 2026-06-20 20:21:57 on dist-test-slave-8z53"
server_key: "0746cabe730f15215d3ee74816106e3a"
server_key_iv: "b29a2934c5a5eb1bd3bdfc91843831d0"
server_key_version: "encryptionkey@0"
I20260620 20:21:57.132931 21962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:57.160463 21962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:57.160931 21962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:57.161051 21962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:57.161541 21962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:57.162278 21962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:57.162321 21962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:57.162355 21962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:57.162377 21962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:57.170701 21962 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:40047
I20260620 20:21:57.170745 22089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:40047 every 8 connection(s)
I20260620 20:21:57.171489 21962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:21:57.175168 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 21962
I20260620 20:21:57.175284 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:21:57.175573 2991 external_mini_cluster.cc:1428] Setting key 2d6ce09459253f0b7714cd623c3a4410
I20260620 20:21:57.180367 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.173017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:33499 (local address 127.2.235.254:37569)
0620 20:21:57.173133 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.173137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.174053 (+ 916us) server_negotiation.cc:407] Connection header received
0620 20:21:57.175632 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.175638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.175720 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.175791 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:57.176839 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.177687 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.178730 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.178938 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.179012 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.179945 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:57.179947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:57.179949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:57.179958 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:57.179972 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:57.179978 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:57.180061 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.180219 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.180263 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:21:57.181098 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.173351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:37569 (local address 127.2.235.195:33499)
0620 20:21:57.173915 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.173949 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.175363 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.175965 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.175972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.176065 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:57.176655 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.176675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.177821 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.177824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.178630 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.178639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.179052 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.179055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.179181 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.179745 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:21:57.179747 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:57.179780 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:21:57.179832 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:57.180093 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:57.180100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:57.180105 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.180201 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.180426 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260620 20:21:57.182111 22090 heartbeater.cc:347] Connected to a master server at 127.2.235.254:37569
I20260620 20:21:57.182331 22090 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:57.182765 22090 heartbeater.cc:511] Master 127.2.235.254:37569 requested a full tablet report, sending...
I20260620 20:21:57.183682 21639 ts_manager.cc:194] Registered new tserver with Master: c094469112a04befa382d7d4376c442d (127.2.235.195:40047)
I20260620 20:21:57.184175 21639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.195:33499
I20260620 20:21:57.189383 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:21:57.194546 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.191197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.2.235.254:37569)
0620 20:21:57.191327 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.191331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.191513 (+ 182us) server_negotiation.cc:407] Connection header received
0620 20:21:57.191616 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.191618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.191650 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.191709 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:57.191945 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.192376 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.193154 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.193385 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.193458 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.194152 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:57.194154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:57.194156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:21:57.194166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:57.194180 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:21:57.194187 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:57.194277 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.194351 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.194376 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:21:57.197290 21639 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:21:57.199916 21639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:21:57.213665 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.207606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38033 (local address 127.0.0.1:52958)
0620 20:21:57.207922 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.207960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.208066 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.208771 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.208777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.208797 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.208913 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.208921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.210326 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.210329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.211901 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.211914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.213194 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.213199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.213318 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.213356 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.213455 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.213549 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260620 20:21:57.214103 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.207743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.2.235.193:38033)
0620 20:21:57.208068 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.208079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.208099 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:21:57.208225 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.208230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.208444 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.208591 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.209039 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.210194 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.212069 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.212899 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.212979 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.213144 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.213861 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.213956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":149,"threads_started":1}
I20260620 20:21:57.215451 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.208251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40047 (local address 127.0.0.1:41920)
0620 20:21:57.208568 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.208601 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.208671 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.210666 (+ 1995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.210670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.210689 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.210818 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.210826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.212629 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.212635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.213775 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.213790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.214933 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.214939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.215084 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.215107 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.215173 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.215325 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":28,"thread_start_us":124,"threads_started":1}
I20260620 20:21:57.215516 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.208418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.2.235.195:40047)
0620 20:21:57.208934 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.208947 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.208971 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:21:57.209037 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.209041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.210045 (+ 1004us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.210470 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.210974 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.212200 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.213965 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.214801 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.214929 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.215108 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.215205 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.215321 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":118,"threads_started":1}
I20260620 20:21:57.217993 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.210322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:42885 (local address 127.0.0.1:44224)
0620 20:21:57.210869 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.210885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.210978 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.213076 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.213079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.213095 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.213168 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.213175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.215027 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.215030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.216604 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.216616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.217576 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.217580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.217688 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.217708 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.217793 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.217855 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":55,"thread_start_us":316,"threads_started":1}
I20260620 20:21:57.218204 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.209928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.2.235.194:42885)
0620 20:21:57.212530 (+ 2602us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.212542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.212565 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:21:57.212633 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.212638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.212798 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.212944 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.213317 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.214908 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.216732 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.217461 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.217578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.217728 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.217816 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.217910 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":115,"threads_started":1}
I20260620 20:21:57.220530 21893 tablet_service.cc:1467] Processing CreateTablet for tablet b08c7b5ad07a407193ecb9333ba062e2 (DEFAULT_TABLE table=test-table [id=289524eb02e54f75864db0c4a33ae28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:57.220530 21762 tablet_service.cc:1467] Processing CreateTablet for tablet b08c7b5ad07a407193ecb9333ba062e2 (DEFAULT_TABLE table=test-table [id=289524eb02e54f75864db0c4a33ae28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:57.221616 21893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08c7b5ad07a407193ecb9333ba062e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:57.221616 21762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08c7b5ad07a407193ecb9333ba062e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:57.226344 22024 tablet_service.cc:1467] Processing CreateTablet for tablet b08c7b5ad07a407193ecb9333ba062e2 (DEFAULT_TABLE table=test-table [id=289524eb02e54f75864db0c4a33ae28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:57.227200 22024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b08c7b5ad07a407193ecb9333ba062e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:57.228834 22107 tablet_bootstrap.cc:492] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: Bootstrap starting.
I20260620 20:21:57.229990 22108 tablet_bootstrap.cc:492] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Bootstrap starting.
I20260620 20:21:57.231115 22107 tablet_bootstrap.cc:654] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:57.231808 22108 tablet_bootstrap.cc:654] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:57.232023 22107 log.cc:824] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:57.232615 22108 log.cc:824] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:57.234012 22107 tablet_bootstrap.cc:492] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: No bootstrap required, opened a new log
I20260620 20:21:57.234141 22111 tablet_bootstrap.cc:492] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Bootstrap starting.
I20260620 20:21:57.234187 22107 ts_tablet_manager.cc:1397] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:21:57.234849 22108 tablet_bootstrap.cc:492] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: No bootstrap required, opened a new log
I20260620 20:21:57.235050 22108 ts_tablet_manager.cc:1397] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:21:57.236112 22111 tablet_bootstrap.cc:654] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:57.236856 22111 log.cc:824] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:57.238372 22107 raft_consensus.cc:348] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.238618 22107 raft_consensus.cc:374] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:57.238678 22107 raft_consensus.cc:729] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915a1f7504bd434197dd7b6338f9d2ac, State: Initialized, Role: FOLLOWER
I20260620 20:21:57.238785 22111 tablet_bootstrap.cc:492] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: No bootstrap required, opened a new log
I20260620 20:21:57.238687 22108 raft_consensus.cc:348] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.238924 22108 raft_consensus.cc:374] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:57.238947 22111 ts_tablet_manager.cc:1397] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:57.238977 22108 raft_consensus.cc:729] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ef6f38ea904626aa00152aeabc832d, State: Initialized, Role: FOLLOWER
I20260620 20:21:57.239126 22107 consensus_queue.cc:260] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.239465 22108 consensus_queue.cc:260] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.239760 21828 heartbeater.cc:503] Master 127.2.235.254:37569 was elected leader, sending a full tablet report...
I20260620 20:21:57.240038 22107 ts_tablet_manager.cc:1428] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 20:21:57.240345 22108 ts_tablet_manager.cc:1428] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:21:57.240404 21959 heartbeater.cc:503] Master 127.2.235.254:37569 was elected leader, sending a full tablet report...
I20260620 20:21:57.242815 22111 raft_consensus.cc:348] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.243046 22111 raft_consensus.cc:374] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:57.243113 22111 raft_consensus.cc:729] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c094469112a04befa382d7d4376c442d, State: Initialized, Role: FOLLOWER
I20260620 20:21:57.243510 22111 consensus_queue.cc:260] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.244132 22090 heartbeater.cc:503] Master 127.2.235.254:37569 was elected leader, sending a full tablet report...
I20260620 20:21:57.244364 22111 ts_tablet_manager.cc:1428] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260620 20:21:57.320293 21829 tablet.cc:2369] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:57.320504 22113 raft_consensus.cc:482] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:57.320644 22113 raft_consensus.cc:504] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.320720 22114 raft_consensus.cc:482] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:57.320884 22114 raft_consensus.cc:504] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.321774 22113 leader_election.cc:288] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c094469112a04befa382d7d4376c442d (127.2.235.195:40047), b3ef6f38ea904626aa00152aeabc832d (127.2.235.194:42885)
I20260620 20:21:57.321830 22114 leader_election.cc:288] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 915a1f7504bd434197dd7b6338f9d2ac (127.2.235.193:38033), c094469112a04befa382d7d4376c442d (127.2.235.195:40047)
I20260620 20:21:57.326399 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.321973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38033 (local address 127.2.235.194:35671)
0620 20:21:57.322131 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.322146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.322211 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.322847 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.322849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.322859 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.322955 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.322962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.323894 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.323897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.325055 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.325085 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.326021 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.326024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.326105 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.326140 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.326200 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.326274 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:21:57.326521 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.322428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:45781 (local address 127.2.235.194:42885)
0620 20:21:57.322516 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.322521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.322575 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:21:57.322717 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.322720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.322768 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.322808 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.323120 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.323806 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.325308 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.325891 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.325963 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.326004 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.326348 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.326413 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":3328}
I20260620 20:21:57.326617 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.321952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40047 (local address 127.2.235.193:41823)
0620 20:21:57.322066 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.322078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.322149 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.322683 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.322687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.322703 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.322819 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.322825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.323805 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.323808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.325380 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.325389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.326275 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.326281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.326374 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.326391 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.326444 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.326500 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":268}
I20260620 20:21:57.326871 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.322399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:35671 (local address 127.2.235.193:38033)
0620 20:21:57.322659 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.322664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.322678 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:21:57.322721 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.322723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.322774 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.322833 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.323068 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.323776 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.325203 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.325836 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.325924 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.325964 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.326654 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.326729 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260620 20:21:57.327055 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.322301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:41823 (local address 127.2.235.195:40047)
0620 20:21:57.322405 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.322409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.322423 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:21:57.322476 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.322480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.322518 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.322582 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.322922 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.323680 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.325501 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.326087 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.326190 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.326225 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.326799 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.326895 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":43}
I20260620 20:21:57.327086 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.322173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:42885 (local address 127.2.235.193:45781)
0620 20:21:57.322513 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.322524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.322627 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.322922 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.322924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.322936 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.323018 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.323023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.323917 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.323919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.325137 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.325148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.326099 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.326103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.326212 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.326233 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.326614 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.326676 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":47,"thread_start_us":102,"threads_started":1}
I20260620 20:21:57.327266 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.322194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:40047 (local address 127.2.235.194:39765)
0620 20:21:57.322656 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.322667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:57.322750 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:57.323300 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:57.323303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:57.323316 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:57.323437 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.323442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.324477 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.324480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.325638 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:57.325647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.326858 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.326862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:57.326966 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.327009 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:21:57.327082 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:21:57.327145 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"spinlock_wait_cycles":6016,"thread_start_us":109,"threads_started":1}
I20260620 20:21:57.327366 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "b3ef6f38ea904626aa00152aeabc832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915a1f7504bd434197dd7b6338f9d2ac" is_pre_election: true
I20260620 20:21:57.327539 21782 raft_consensus.cc:2454] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ef6f38ea904626aa00152aeabc832d in term 0.
I20260620 20:21:57.327557 21913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "915a1f7504bd434197dd7b6338f9d2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ef6f38ea904626aa00152aeabc832d" is_pre_election: true
I20260620 20:21:57.327617 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.322442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:39765 (local address 127.2.235.195:40047)
0620 20:21:57.323042 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.323051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.323062 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:21:57.323094 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.323097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.323138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.323201 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:57.323593 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.324346 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.325788 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.326336 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.326434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.327368 (+ 934us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.327405 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.327469 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":92,"threads_started":1}
I20260620 20:21:57.327702 21913 raft_consensus.cc:2454] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915a1f7504bd434197dd7b6338f9d2ac in term 0.
I20260620 20:21:57.327916 21848 leader_election.cc:302] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915a1f7504bd434197dd7b6338f9d2ac, b3ef6f38ea904626aa00152aeabc832d; no voters:
I20260620 20:21:57.328054 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "b3ef6f38ea904626aa00152aeabc832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094469112a04befa382d7d4376c442d" is_pre_election: true
I20260620 20:21:57.328308 21717 leader_election.cc:302] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915a1f7504bd434197dd7b6338f9d2ac, b3ef6f38ea904626aa00152aeabc832d; no voters:
I20260620 20:21:57.328322 22044 raft_consensus.cc:2454] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ef6f38ea904626aa00152aeabc832d in term 0.
I20260620 20:21:57.328357 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "915a1f7504bd434197dd7b6338f9d2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094469112a04befa382d7d4376c442d" is_pre_election: true
I20260620 20:21:57.328434 22043 raft_consensus.cc:2454] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915a1f7504bd434197dd7b6338f9d2ac in term 0.
I20260620 20:21:57.328449 22114 raft_consensus.cc:2781] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:57.328516 22114 raft_consensus.cc:482] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:57.328529 22113 raft_consensus.cc:2781] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:57.328559 22114 raft_consensus.cc:3037] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:57.328589 22113 raft_consensus.cc:482] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:57.328629 22113 raft_consensus.cc:3037] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:57.329794 22114 raft_consensus.cc:504] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.329792 22113 raft_consensus.cc:504] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.330099 22113 leader_election.cc:288] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [CANDIDATE]: Term 1 election: Requested vote from peers c094469112a04befa382d7d4376c442d (127.2.235.195:40047), b3ef6f38ea904626aa00152aeabc832d (127.2.235.194:42885)
I20260620 20:21:57.330099 22114 leader_election.cc:288] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [CANDIDATE]: Term 1 election: Requested vote from peers 915a1f7504bd434197dd7b6338f9d2ac (127.2.235.193:38033), c094469112a04befa382d7d4376c442d (127.2.235.195:40047)
I20260620 20:21:57.330426 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "915a1f7504bd434197dd7b6338f9d2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094469112a04befa382d7d4376c442d"
I20260620 20:21:57.330442 21913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "915a1f7504bd434197dd7b6338f9d2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ef6f38ea904626aa00152aeabc832d"
I20260620 20:21:57.330484 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "b3ef6f38ea904626aa00152aeabc832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094469112a04befa382d7d4376c442d"
I20260620 20:21:57.330484 21782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b08c7b5ad07a407193ecb9333ba062e2" candidate_uuid: "b3ef6f38ea904626aa00152aeabc832d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915a1f7504bd434197dd7b6338f9d2ac"
I20260620 20:21:57.330596 21913 raft_consensus.cc:2379] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 915a1f7504bd434197dd7b6338f9d2ac in current term 1: Already voted for candidate b3ef6f38ea904626aa00152aeabc832d in this term.
I20260620 20:21:57.330544 22043 raft_consensus.cc:3037] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:57.330650 21782 raft_consensus.cc:2379] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3ef6f38ea904626aa00152aeabc832d in current term 1: Already voted for candidate 915a1f7504bd434197dd7b6338f9d2ac in this term.
I20260620 20:21:57.331082 21848 leader_election.cc:302] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3ef6f38ea904626aa00152aeabc832d; no voters: 915a1f7504bd434197dd7b6338f9d2ac, c094469112a04befa382d7d4376c442d
I20260620 20:21:57.331252 22114 raft_consensus.cc:2726] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 20:21:57.331626 22043 raft_consensus.cc:2454] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915a1f7504bd434197dd7b6338f9d2ac in term 1.
I20260620 20:21:57.331930 21717 leader_election.cc:302] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 915a1f7504bd434197dd7b6338f9d2ac, c094469112a04befa382d7d4376c442d; no voters: b3ef6f38ea904626aa00152aeabc832d
I20260620 20:21:57.332080 22113 raft_consensus.cc:2781] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:57.332273 22113 raft_consensus.cc:686] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [term 1 LEADER]: Becoming Leader. State: Replica: 915a1f7504bd434197dd7b6338f9d2ac, State: Running, Role: LEADER
I20260620 20:21:57.332597 22113 consensus_queue.cc:237] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } }
I20260620 20:21:57.335213 21639 catalog_manager.cc:5495] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 915a1f7504bd434197dd7b6338f9d2ac (127.2.235.193). New cstate: current_term: 1 leader_uuid: "915a1f7504bd434197dd7b6338f9d2ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915a1f7504bd434197dd7b6338f9d2ac" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 38033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 } health_report { overall_health: UNKNOWN } } }
I20260620 20:21:57.364614 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.361542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.2.235.193:38033)
0620 20:21:57.361654 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:57.361657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:57.361698 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:21:57.361809 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:57.361811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:57.361857 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:57.361939 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:57.362163 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.362598 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.363331 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:57.363519 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:57.363624 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:57.363923 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:57.364267 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:57.364373 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:57.364464 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 20:21:57.364507 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:21:57.369730 22043 raft_consensus.cc:1261] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 1 FOLLOWER]: Refusing update from remote peer 915a1f7504bd434197dd7b6338f9d2ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:57.369731 21913 raft_consensus.cc:1261] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 1 FOLLOWER]: Refusing update from remote peer 915a1f7504bd434197dd7b6338f9d2ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:57.370239 22113 consensus_queue.cc:1035] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ef6f38ea904626aa00152aeabc832d" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:57.370414 22119 consensus_queue.cc:1035] T b08c7b5ad07a407193ecb9333ba062e2 P 915a1f7504bd434197dd7b6338f9d2ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c094469112a04befa382d7d4376c442d" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:57.377930 22123 mvcc.cc:205] Tried to move back new op lower bound from 7299018413539983360 to 7299018413396451328. Current Snapshot: MvccSnapshot[applied={T|T < 7299018413539983360}]
I20260620 20:21:57.380461 22124 mvcc.cc:205] Tried to move back new op lower bound from 7299018413539983360 to 7299018413396451328. Current Snapshot: MvccSnapshot[applied={T|T < 7299018413539983360}]
I20260620 20:21:57.385185 21638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:57.385363 21638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:57.387447 21638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1ae3824e054015b4662c89be95edff: Sending DeleteTablet for 3 replicas of tablet b08c7b5ad07a407193ecb9333ba062e2
I20260620 20:21:57.388046 21762 tablet_service.cc:1514] Processing DeleteTablet for tablet b08c7b5ad07a407193ecb9333ba062e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:57 UTC) from {username='slave'} at 127.0.0.1:52958
I20260620 20:21:57.388283 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet b08c7b5ad07a407193ecb9333ba062e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:57 UTC) from {username='slave'} at 127.0.0.1:41920
I20260620 20:21:57.388634 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21700
I20260620 20:21:57.388688 21893 tablet_service.cc:1514] Processing DeleteTablet for tablet b08c7b5ad07a407193ecb9333ba062e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:57 UTC) from {username='slave'} at 127.0.0.1:44224
I20260620 20:21:57.389214 22133 tablet_replica.cc:330] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: stopping tablet replica
I20260620 20:21:57.389493 22133 raft_consensus.cc:2229] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:57.389770 22133 raft_consensus.cc:2258] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:57.391012 22133 ts_tablet_manager.cc:1905] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:57.393419 22132 tablet_replica.cc:330] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: stopping tablet replica
I20260620 20:21:57.393664 22132 raft_consensus.cc:2229] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:57.393925 22132 raft_consensus.cc:2258] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:57.394021 22133 ts_tablet_manager.cc:1918] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:57.394115 22133 log.cc:1196] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/b08c7b5ad07a407193ecb9333ba062e2
I20260620 20:21:57.394419 22133 ts_tablet_manager.cc:1939] T b08c7b5ad07a407193ecb9333ba062e2 P b3ef6f38ea904626aa00152aeabc832d: Deleting consensus metadata
I20260620 20:21:57.395103 22132 ts_tablet_manager.cc:1905] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:57.395553 21625 catalog_manager.cc:4841] TS b3ef6f38ea904626aa00152aeabc832d (127.2.235.194:42885): tablet b08c7b5ad07a407193ecb9333ba062e2 (table test-table [id=289524eb02e54f75864db0c4a33ae28e]) successfully deleted
I20260620 20:21:57.397130 22132 ts_tablet_manager.cc:1918] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:57.397208 22132 log.cc:1196] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/b08c7b5ad07a407193ecb9333ba062e2
I20260620 20:21:57.397294 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21831
W20260620 20:21:57.397339 21625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:38033 (error 108)
I20260620 20:21:57.398128 22132 ts_tablet_manager.cc:1939] T b08c7b5ad07a407193ecb9333ba062e2 P c094469112a04befa382d7d4376c442d: Deleting consensus metadata
I20260620 20:21:57.398993 21625 catalog_manager.cc:4841] TS c094469112a04befa382d7d4376c442d (127.2.235.195:40047): tablet b08c7b5ad07a407193ecb9333ba062e2 (table test-table [id=289524eb02e54f75864db0c4a33ae28e]) successfully deleted
I20260620 20:21:57.401932 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:57.401678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:38033 (local address 127.0.0.1:52978)
0620 20:21:57.401824 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:57.401876 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:38033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 20:21:57.402056 21625 catalog_manager.cc:4584] TS 915a1f7504bd434197dd7b6338f9d2ac (127.2.235.193:38033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b08c7b5ad07a407193ecb9333ba062e2: Network error: Client connection negotiation failed: client connection to 127.2.235.193:38033: connect: Connection refused (error 111)
I20260620 20:21:57.403380 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21962
I20260620 20:21:57.409053 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 21608
2026-06-20T20:21:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1019 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:57 dist-test-slave-8z53 krb5kdc[22137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:57 dist-test-slave-8z53 krb5kdc[22137](info): set up 2 sockets
Jun 20 20:21:57 dist-test-slave-8z53 krb5kdc[22137](info): commencing operation
krb5kdc: starting...
W20260620 20:21:58.381601 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:21:58 dist-test-slave-8z53 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:21:58.487885 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:46785
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:58.589751 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:58.589988 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:58.590030 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:58.594373 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:58.594441 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:58.594470 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:58.594485 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:58.594501 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:58.599668 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:46785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:46785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:58.600559 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:58.601485 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:58.609602 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.609609 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:58.609824 22150 server_base.cc:1034] running on GCE node
W20260620 20:21:58.609611 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:58.610457 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:58.611491 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:58.612674 22150 hybrid_clock.cc:648] HybridClock initialized: now 1781986918612646 us; error 49 us; skew 500 ppm
Jun 20 20:21:58 dist-test-slave-8z53 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:58.615483 22150 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:21:58.616334 22150 webserver.cc:460] Webserver started at http://127.2.235.254:39105/ using document root <none> and password file <none>
I20260620 20:21:58.616852 22150 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:58.616938 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:58.617153 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:58.619084 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "01e78dce6edf4815a6c8b4061fa4d017"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-8z53"
server_key: "f540bad094a59640a8c42c0812f5349b"
server_key_iv: "57c2c04f17043c63b2c026f1cf37b7eb"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.619637 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "01e78dce6edf4815a6c8b4061fa4d017"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-8z53"
server_key: "f540bad094a59640a8c42c0812f5349b"
server_key_iv: "57c2c04f17043c63b2c026f1cf37b7eb"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.623100 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:21:58.625380 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:58.626515 22150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:58.626639 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "01e78dce6edf4815a6c8b4061fa4d017"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-8z53"
server_key: "f540bad094a59640a8c42c0812f5349b"
server_key_iv: "57c2c04f17043c63b2c026f1cf37b7eb"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.626737 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:58.644155 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:58.647125 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:58.647300 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:58.654274 22150 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:46785
I20260620 20:21:58.654300 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:46785 every 8 connection(s)
I20260620 20:21:58.655123 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:58.657662 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:58.662513 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22150
I20260620 20:21:58.662662 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:21:58.662936 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017: Bootstrap starting.
I20260620 20:21:58.663039 2991 external_mini_cluster.cc:1428] Setting key df6a90fabe8fbc6a82ee062238df1eb1
I20260620 20:21:58.665464 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:58.666178 22218 log.cc:824] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:58.668097 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017: No bootstrap required, opened a new log
Jun 20 20:21:58 dist-test-slave-8z53 krb5kdc[22137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:58.671680 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e78dce6edf4815a6c8b4061fa4d017" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46785 } }
I20260620 20:21:58.671875 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:58.671952 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e78dce6edf4815a6c8b4061fa4d017, State: Initialized, Role: FOLLOWER
I20260620 20:21:58.672402 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e78dce6edf4815a6c8b4061fa4d017" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46785 } }
I20260620 20:21:58.672556 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:58.672621 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:58.672719 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:58.673659 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e78dce6edf4815a6c8b4061fa4d017" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46785 } }
I20260620 20:21:58.673981 22218 leader_election.cc:302] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e78dce6edf4815a6c8b4061fa4d017; no voters:
I20260620 20:21:58.674294 22218 leader_election.cc:288] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:58.674350 22223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:58.674536 22223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [term 1 LEADER]: Becoming Leader. State: Replica: 01e78dce6edf4815a6c8b4061fa4d017, State: Running, Role: LEADER
I20260620 20:21:58.674821 22223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e78dce6edf4815a6c8b4061fa4d017" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46785 } }
I20260620 20:21:58.675493 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:21:58.675999 22225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e78dce6edf4815a6c8b4061fa4d017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e78dce6edf4815a6c8b4061fa4d017" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46785 } } }
I20260620 20:21:58.676759 22225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:58.676635 22224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e78dce6edf4815a6c8b4061fa4d017. Latest consensus state: current_term: 1 leader_uuid: "01e78dce6edf4815a6c8b4061fa4d017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e78dce6edf4815a6c8b4061fa4d017" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 46785 } } }
I20260620 20:21:58.677073 22224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:58.677217 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:58.664363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.2.235.254:46785)
0620 20:21:58.664762 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:58.664775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:58.664815 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:21:58.665461 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:58.665477 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:58.665819 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:58.666126 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:58.666887 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.667821 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.668726 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.669174 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.669387 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:58.671893 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:58.671900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:58.671917 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:58.671970 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:58.673863 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:58.674312 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:58.674317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:58.674322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:58.674450 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:58.674772 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:58.674777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:58.674782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:58.675262 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:58.675482 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:58.676062 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 20:21:58.676344 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
I20260620 20:21:58.677675 22232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:58.680634 22232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:58.685176 22232 catalog_manager.cc:1347] Generated new cluster ID: d6173b0e090546ae85fce52b01e180c5
I20260620 20:21:58.685240 22232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:58.692813 22232 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:58.693854 22232 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:58.703722 22232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017: Generated new TSK 0
I20260620 20:21:58.704320 22232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.235.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
Entry for principal kudu/127.2.235.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab.
I20260620 20:21:58.732124 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46785
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:58.833586 22244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:58.833819 22244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:58.833896 22244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:58.838476 22244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:58.838529 22244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:58.838727 22244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:21:58.843652 22244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.193.keytab
--principal=kudu/127.2.235.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:58.844457 22244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:58.845429 22244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:58.853807 22250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.853829 22249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.853832 22252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:58.853879 22244 server_base.cc:1034] running on GCE node
I20260620 20:21:58.854355 22244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:58.854969 22244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:58.856132 22244 hybrid_clock.cc:648] HybridClock initialized: now 1781986918856110 us; error 32 us; skew 500 ppm
Jun 20 20:21:58 dist-test-slave-8z53 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:58.858713 22244 init.cc:377] Logged in from keytab as kudu/127.2.235.193@KRBTEST.COM (short username kudu)
I20260620 20:21:58.859473 22244 webserver.cc:460] Webserver started at http://127.2.235.193:43735/ using document root <none> and password file <none>
I20260620 20:21:58.859905 22244 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:58.859962 22244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:58.860123 22244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:58.861922 22244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "24823002ca7a42eda21a575d14481a62"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-8z53"
server_key: "35223abd9d89954311b1acee3970ca49"
server_key_iv: "47a120fc2099a3fada9557c318c99616"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.862414 22244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "24823002ca7a42eda21a575d14481a62"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-8z53"
server_key: "35223abd9d89954311b1acee3970ca49"
server_key_iv: "47a120fc2099a3fada9557c318c99616"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.865765 22244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:58.867970 22259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:58.869060 22244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:58.869203 22244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "24823002ca7a42eda21a575d14481a62"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-8z53"
server_key: "35223abd9d89954311b1acee3970ca49"
server_key_iv: "47a120fc2099a3fada9557c318c99616"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.869369 22244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:58.891674 22244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:58.894732 22244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:58.894930 22244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:58.895491 22244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:58.896277 22244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:58.896345 22244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:58.896407 22244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:58.896457 22244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:58.905444 22244 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:44383
I20260620 20:21:58.905489 22372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:44383 every 8 connection(s)
I20260620 20:21:58.906265 22244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:58.907501 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22244
I20260620 20:21:58.907691 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:21:58.908037 2991 external_mini_cluster.cc:1428] Setting key 1f081097b7a3bf693b9b86c4135ae063
Jun 20 20:21:58 dist-test-slave-8z53 krb5kdc[22137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.193@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.194@KRBTEST.COM" created.
I20260620 20:21:58.923103 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:58.907832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:55543 (local address 127.2.235.254:46785)
0620 20:21:58.907953 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:58.907958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:58.908930 (+ 972us) server_negotiation.cc:407] Connection header received
0620 20:21:58.910071 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:58.910074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:58.910116 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:58.910184 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:58.911695 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.912564 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.914377 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.914837 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.914935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:58.918212 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:58.918217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:58.918221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:58.918248 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:58.920669 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:58.921381 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:58.921384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:58.921386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:58.921476 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:58.921954 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:58.921960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:58.921962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:58.922212 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:58.922323 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:58.922816 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 20:21:58.922936 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:21:58.923767 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:58.908224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:46785 (local address 127.2.235.193:55543)
0620 20:21:58.908776 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:58.908829 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:58.909837 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:58.910378 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:58.910389 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:58.910896 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:58.911520 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:58.911537 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.912714 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.912726 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:58.914211 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:58.914225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.915160 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.915167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:58.915389 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:58.916089 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:58.916112 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:58.917982 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:58.920783 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:58.920788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:58.920800 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:58.921287 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:58.921600 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:58.921604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:58.921608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:58.921788 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:58.922324 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:58.922331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:58.922566 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 20:21:58.922755 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:21:58.923029 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":144,"threads_started":1}
I20260620 20:21:58.924943 22373 heartbeater.cc:347] Connected to a master server at 127.2.235.254:46785
I20260620 20:21:58.925155 22373 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:58.925616 22373 heartbeater.cc:511] Master 127.2.235.254:46785 requested a full tablet report, sending...
I20260620 20:21:58.927043 22182 ts_manager.cc:194] Registered new tserver with Master: 24823002ca7a42eda21a575d14481a62 (127.2.235.193:44383)
I20260620 20:21:58.928053 22182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.193@KRBTEST.COM'} at 127.2.235.193:55543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
Entry for principal kudu/127.2.235.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab.
I20260620 20:21:58.936267 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46785
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:59.035445 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:59.035670 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:59.035709 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:59.039415 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:59.039458 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:59.039541 22378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:21:59.044024 22378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.194.keytab
--principal=kudu/127.2.235.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:59.044852 22378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:59.045764 22378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:59.053911 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.053932 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.053932 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:59.054227 22378 server_base.cc:1034] running on GCE node
I20260620 20:21:59.054566 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:59.055064 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:59.056219 22378 hybrid_clock.cc:648] HybridClock initialized: now 1781986919056206 us; error 37 us; skew 500 ppm
Jun 20 20:21:59 dist-test-slave-8z53 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:59.058684 22378 init.cc:377] Logged in from keytab as kudu/127.2.235.194@KRBTEST.COM (short username kudu)
I20260620 20:21:59.059386 22378 webserver.cc:460] Webserver started at http://127.2.235.194:41663/ using document root <none> and password file <none>
I20260620 20:21:59.059814 22378 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:59.059867 22378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:59.060002 22378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:59.061786 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "347e99631f9f4497bb50f65d01665ece"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-8z53"
server_key: "48168be741332e1d61c25a6b92031b79"
server_key_iv: "2585c6bb556c821040b7bfa55d3e7099"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.062273 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "347e99631f9f4497bb50f65d01665ece"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-8z53"
server_key: "48168be741332e1d61c25a6b92031b79"
server_key_iv: "2585c6bb556c821040b7bfa55d3e7099"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.065507 22378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:59.067549 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.068634 22378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:21:59.068789 22378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "347e99631f9f4497bb50f65d01665ece"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-8z53"
server_key: "48168be741332e1d61c25a6b92031b79"
server_key_iv: "2585c6bb556c821040b7bfa55d3e7099"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.068897 22378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:59.087129 22378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:59.090034 22378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:59.090209 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:59.090711 22378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:59.091418 22378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:59.091485 22378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.091545 22378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:59.091573 22378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.099852 22378 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:46787
I20260620 20:21:59.099885 22506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:46787 every 8 connection(s)
I20260620 20:21:59.100586 22378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:59.101074 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22378
I20260620 20:21:59.101202 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:21:59.101527 2991 external_mini_cluster.cc:1428] Setting key 623ca1cd6b1904374be87041b8293153
Jun 20 20:21:59 dist-test-slave-8z53 krb5kdc[22137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.194@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.235.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.195@KRBTEST.COM" created.
I20260620 20:21:59.114729 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.102314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:35411 (local address 127.2.235.254:46785)
0620 20:21:59.102457 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.102462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.103300 (+ 838us) server_negotiation.cc:407] Connection header received
0620 20:21:59.104573 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.104575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.104618 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.104685 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:59.106227 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.106733 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.108483 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.108694 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.108780 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.111307 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:59.111311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:59.111312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:59.111334 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:59.112735 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.113212 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.113217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.113223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.113338 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.113676 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.113682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.113685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.113882 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:59.113983 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.114501 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.114600 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:21:59.115726 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.102644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:46785 (local address 127.2.235.194:35411)
0620 20:21:59.103140 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.103191 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.104307 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.104898 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.104907 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.105429 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:59.106067 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.106086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.106859 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.106865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.108348 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.108364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.108848 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.108853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.109072 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.109621 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:59.109645 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:59.111124 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:59.112850 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.112856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.112868 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.113102 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.113453 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.113456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.113458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.113580 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.113990 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:59.113999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:59.114249 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.114446 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.114808 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":151,"threads_started":1}
I20260620 20:21:59.116773 22507 heartbeater.cc:347] Connected to a master server at 127.2.235.254:46785
I20260620 20:21:59.117000 22507 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:59.117544 22507 heartbeater.cc:511] Master 127.2.235.254:46785 requested a full tablet report, sending...
I20260620 20:21:59.118589 22182 ts_manager.cc:194] Registered new tserver with Master: 347e99631f9f4497bb50f65d01665ece (127.2.235.194:46787)
I20260620 20:21:59.119095 22182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.194@KRBTEST.COM'} at 127.2.235.194:35411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
Entry for principal kudu/127.2.235.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab.
I20260620 20:21:59.127604 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46785
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:59.225889 22512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:59.226110 22512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:59.226147 22512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:59.229949 22512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:59.229995 22512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:59.230069 22512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:21:59.234720 22512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.195.keytab
--principal=kudu/127.2.235.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:59.235569 22512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:59.236430 22512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:59.244650 22518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:59.244736 22512 server_base.cc:1034] running on GCE node
W20260620 20:21:59.244621 22517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.244622 22520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:59.245314 22512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:59.245865 22512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:59.247061 22512 hybrid_clock.cc:648] HybridClock initialized: now 1781986919247050 us; error 31 us; skew 500 ppm
Jun 20 20:21:59 dist-test-slave-8z53 krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:59.249512 22512 init.cc:377] Logged in from keytab as kudu/127.2.235.195@KRBTEST.COM (short username kudu)
I20260620 20:21:59.250293 22512 webserver.cc:460] Webserver started at http://127.2.235.195:46879/ using document root <none> and password file <none>
I20260620 20:21:59.250754 22512 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:59.250842 22512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:59.251049 22512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:59.252774 22512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "920b396eadfb4a918f207a25d277316f"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-8z53"
server_key: "f9f43b01e26d30b92a0c39c06361e5a8"
server_key_iv: "24ee64474b92341f94254f1382c12d18"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.253321 22512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "920b396eadfb4a918f207a25d277316f"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-8z53"
server_key: "f9f43b01e26d30b92a0c39c06361e5a8"
server_key_iv: "24ee64474b92341f94254f1382c12d18"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.256531 22512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:21:59.258661 22527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.259706 22512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 20:21:59.259824 22512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "920b396eadfb4a918f207a25d277316f"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-8z53"
server_key: "f9f43b01e26d30b92a0c39c06361e5a8"
server_key_iv: "24ee64474b92341f94254f1382c12d18"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.259924 22512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:59.289369 22512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:59.291728 22512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:59.291903 22512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:59.292443 22512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:59.293193 22512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:59.293279 22512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.293368 22512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:59.293423 22512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.301712 22512 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:39691
I20260620 20:21:59.301736 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:39691 every 8 connection(s)
I20260620 20:21:59.302484 22512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
Jun 20 20:21:59 dist-test-slave-8z53 krb5kdc[22137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.195@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:21:59.312706 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22512
I20260620 20:21:59.312844 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:21:59.313134 2991 external_mini_cluster.cc:1428] Setting key d3de112bc8471a93002613ea494bcf82
I20260620 20:21:59.314889 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.304114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:53295 (local address 127.2.235.254:46785)
0620 20:21:59.304255 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.304258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.304978 (+ 720us) server_negotiation.cc:407] Connection header received
0620 20:21:59.306181 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.306184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.306240 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.306316 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:59.307327 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.307877 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.309065 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.309356 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.309467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.311500 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:59.311503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:59.311506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:59.311528 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:59.312692 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.313225 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.313230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.313231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.313341 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.313714 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.313717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.313718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.313969 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:59.314072 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.314619 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.314745 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:21:59.315631 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.304443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:46785 (local address 127.2.235.195:53295)
0620 20:21:59.304830 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.304867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.305981 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.306476 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.306482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.306777 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:59.307182 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.307194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.308000 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.308004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.308940 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.308950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.309498 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.309501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.309660 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.310070 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:59.310084 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:59.311352 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:59.312806 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.312811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.312824 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.313125 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.313446 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.313449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.313451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.313610 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.314079 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:59.314085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:59.314361 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.314558 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.314796 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
I20260620 20:21:59.316569 22641 heartbeater.cc:347] Connected to a master server at 127.2.235.254:46785
I20260620 20:21:59.316782 22641 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:59.317200 22641 heartbeater.cc:511] Master 127.2.235.254:46785 requested a full tablet report, sending...
I20260620 20:21:59.318127 22182 ts_manager.cc:194] Registered new tserver with Master: 920b396eadfb4a918f207a25d277316f (127.2.235.195:39691)
I20260620 20:21:59.318666 22182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.235.195@KRBTEST.COM'} at 127.2.235.195:53295
I20260620 20:21:59.326926 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:21:59.335778 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.328705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.2.235.254:46785)
0620 20:21:59.328830 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.328835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.329025 (+ 190us) server_negotiation.cc:407] Connection header received
0620 20:21:59.329189 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.329192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.329293 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.329386 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:59.330021 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.330534 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.331341 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.331547 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.331663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.332897 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:59.332901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:59.332903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:59.332952 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:59.334152 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.334556 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.334561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.334563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.334639 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.334917 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.334920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.334922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.335117 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:59.335254 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.335519 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.335625 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:21:59.338835 22182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:21:59.341759 22182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:21:59.355063 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.348937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.2.235.195:39691)
0620 20:21:59.349769 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.349779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.349798 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:21:59.349850 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.349854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.349981 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.350119 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:59.350742 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.351910 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.353826 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.354324 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.354457 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.354585 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.354794 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.354892 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":83,"threads_started":1}
I20260620 20:21:59.355262 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.348686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:44383 (local address 127.0.0.1:45104)
0620 20:21:59.349344 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.349384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.349531 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.350269 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.350275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.350305 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:59.350436 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.350443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.352103 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.352107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.353449 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.353458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.354707 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.354711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.354807 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.354884 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.354969 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.355105 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":136,"spinlock_wait_cycles":7296,"thread_start_us":71,"threads_started":1}
I20260620 20:21:59.355254 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.348796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.2.235.193:44383)
0620 20:21:59.349105 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.349116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.349486 (+ 370us) server_negotiation.cc:407] Connection header received
0620 20:21:59.349754 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.349760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.349966 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.350119 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:59.350824 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.351944 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.353575 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.354356 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.354475 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.354608 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.355004 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.355105 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":147,"threads_started":1}
I20260620 20:21:59.355510 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.349144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:39691 (local address 127.0.0.1:58748)
0620 20:21:59.349387 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.349400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.349529 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.350269 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.350275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.350303 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:59.350643 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.350650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.352070 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.352074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.353682 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.353694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.354459 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.354462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.354562 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.354596 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.355245 (+ 649us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.355321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":356,"spinlock_wait_cycles":9728,"thread_start_us":103,"threads_started":1}
I20260620 20:21:59.357744 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.350086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.2.235.194:46787)
0620 20:21:59.350505 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.350519 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.350539 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:21:59.350689 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.350696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.350889 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.351029 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:59.351211 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.351955 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.353819 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.354596 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.354751 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.354887 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.355106 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.357595 (+ 2489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260620 20:21:59.358062 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.349749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:46787 (local address 127.0.0.1:44028)
0620 20:21:59.350461 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.350476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.350559 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.351016 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.351019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.351035 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:59.351121 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.351127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.352106 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.352108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.353682 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.353694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.354737 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.354743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.354884 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.354907 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.354977 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.357935 (+ 2958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":3117,"thread_start_us":94,"threads_started":1}
I20260620 20:21:59.357961 22307 tablet_service.cc:1467] Processing CreateTablet for tablet 3e389bbd790f4795bc3f2cc4a78d3c81 (DEFAULT_TABLE table=test-table [id=e102e3a416e64ff9993f194e72f527fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:59.358084 22575 tablet_service.cc:1467] Processing CreateTablet for tablet 3e389bbd790f4795bc3f2cc4a78d3c81 (DEFAULT_TABLE table=test-table [id=e102e3a416e64ff9993f194e72f527fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:59.358950 22307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e389bbd790f4795bc3f2cc4a78d3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:59.358999 22575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e389bbd790f4795bc3f2cc4a78d3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:59.365664 22658 tablet_bootstrap.cc:492] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Bootstrap starting.
I20260620 20:21:59.365545 22441 tablet_service.cc:1467] Processing CreateTablet for tablet 3e389bbd790f4795bc3f2cc4a78d3c81 (DEFAULT_TABLE table=test-table [id=e102e3a416e64ff9993f194e72f527fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:21:59.366060 22659 tablet_bootstrap.cc:492] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: Bootstrap starting.
I20260620 20:21:59.366484 22441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e389bbd790f4795bc3f2cc4a78d3c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:59.367707 22658 tablet_bootstrap.cc:654] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:59.367978 22659 tablet_bootstrap.cc:654] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:59.368592 22658 log.cc:824] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:59.369154 22659 log.cc:824] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:59.370465 22658 tablet_bootstrap.cc:492] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: No bootstrap required, opened a new log
I20260620 20:21:59.370656 22658 ts_tablet_manager.cc:1397] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:59.370992 22659 tablet_bootstrap.cc:492] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: No bootstrap required, opened a new log
I20260620 20:21:59.371181 22659 ts_tablet_manager.cc:1397] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:59.374212 22658 raft_consensus.cc:348] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.374440 22658 raft_consensus.cc:374] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:59.374500 22658 raft_consensus.cc:729] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920b396eadfb4a918f207a25d277316f, State: Initialized, Role: FOLLOWER
I20260620 20:21:59.374531 22659 raft_consensus.cc:348] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.374776 22659 raft_consensus.cc:374] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:59.374854 22659 raft_consensus.cc:729] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24823002ca7a42eda21a575d14481a62, State: Initialized, Role: FOLLOWER
I20260620 20:21:59.374933 22658 consensus_queue.cc:260] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.375293 22659 consensus_queue.cc:260] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.375674 22641 heartbeater.cc:503] Master 127.2.235.254:46785 was elected leader, sending a full tablet report...
I20260620 20:21:59.375954 22373 heartbeater.cc:503] Master 127.2.235.254:46785 was elected leader, sending a full tablet report...
I20260620 20:21:59.376021 22658 ts_tablet_manager.cc:1428] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:21:59.376991 22659 ts_tablet_manager.cc:1428] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:21:59.377468 22662 tablet_bootstrap.cc:492] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Bootstrap starting.
I20260620 20:21:59.379684 22662 tablet_bootstrap.cc:654] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:59.380537 22662 log.cc:824] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:59.382052 22662 tablet_bootstrap.cc:492] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: No bootstrap required, opened a new log
I20260620 20:21:59.382234 22662 ts_tablet_manager.cc:1397] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:21:59.384665 22662 raft_consensus.cc:348] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.384894 22662 raft_consensus.cc:374] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:59.384951 22662 raft_consensus.cc:729] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347e99631f9f4497bb50f65d01665ece, State: Initialized, Role: FOLLOWER
I20260620 20:21:59.385334 22662 consensus_queue.cc:260] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.385948 22507 heartbeater.cc:503] Master 127.2.235.254:46785 was elected leader, sending a full tablet report...
I20260620 20:21:59.386142 22662 ts_tablet_manager.cc:1428] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260620 20:21:59.407315 22374 tablet.cc:2369] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:21:59.553476 22642 tablet.cc:2369] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:21:59.601576 22508 tablet.cc:2369] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:21:59.714787 22664 raft_consensus.cc:482] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:21:59.714968 22664 raft_consensus.cc:504] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.716030 22664 leader_election.cc:288] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 920b396eadfb4a918f207a25d277316f (127.2.235.195:39691), 347e99631f9f4497bb50f65d01665ece (127.2.235.194:46787)
I20260620 20:21:59.720829 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.716354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:50835 (local address 127.2.235.195:39691)
0620 20:21:59.716507 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.716512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.716643 (+ 131us) server_negotiation.cc:407] Connection header received
0620 20:21:59.716796 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.716798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.716845 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.716919 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:59.717237 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.717987 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.719674 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.720225 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.720339 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.720387 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.720655 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.720715 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:21:59.720829 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.716368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:37035 (local address 127.2.235.194:46787)
0620 20:21:59.716561 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.716564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.716688 (+ 124us) server_negotiation.cc:407] Connection header received
0620 20:21:59.716764 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.716768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.716819 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.716912 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:21:59.717267 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.717973 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.719674 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.720225 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.720339 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.720387 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.720655 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.720716 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 20:21:59.720877 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.716212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:39691 (local address 127.2.235.193:50835)
0620 20:21:59.716549 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.716564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.716677 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.716951 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.716953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.716969 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:59.717124 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.717131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.718099 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.718102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.719546 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.719557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.720344 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.720347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.720498 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.720552 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.720626 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.720724 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":211,"spinlock_wait_cycles":384,"thread_start_us":77,"threads_started":1}
I20260620 20:21:59.720880 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.716233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:46787 (local address 127.2.235.193:37035)
0620 20:21:59.716614 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.716625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.716686 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.716951 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.716953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.716991 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:21:59.717136 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.717140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.718099 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.718102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.719550 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.719557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.720344 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.720347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.720507 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.720544 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.720626 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.720726 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":285,"spinlock_wait_cycles":29440,"thread_start_us":107,"threads_started":1}
I20260620 20:21:59.721544 22461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e389bbd790f4795bc3f2cc4a78d3c81" candidate_uuid: "24823002ca7a42eda21a575d14481a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347e99631f9f4497bb50f65d01665ece" is_pre_election: true
I20260620 20:21:59.721581 22595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e389bbd790f4795bc3f2cc4a78d3c81" candidate_uuid: "24823002ca7a42eda21a575d14481a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920b396eadfb4a918f207a25d277316f" is_pre_election: true
I20260620 20:21:59.721808 22461 raft_consensus.cc:2454] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24823002ca7a42eda21a575d14481a62 in term 0.
I20260620 20:21:59.721850 22595 raft_consensus.cc:2454] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24823002ca7a42eda21a575d14481a62 in term 0.
I20260620 20:21:59.722229 22262 leader_election.cc:302] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24823002ca7a42eda21a575d14481a62, 347e99631f9f4497bb50f65d01665ece; no voters:
I20260620 20:21:59.722466 22664 raft_consensus.cc:2781] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:21:59.722537 22664 raft_consensus.cc:482] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:21:59.722599 22664 raft_consensus.cc:3037] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:59.723564 22664 raft_consensus.cc:504] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.723914 22664 leader_election.cc:288] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [CANDIDATE]: Term 1 election: Requested vote from peers 920b396eadfb4a918f207a25d277316f (127.2.235.195:39691), 347e99631f9f4497bb50f65d01665ece (127.2.235.194:46787)
I20260620 20:21:59.724222 22595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e389bbd790f4795bc3f2cc4a78d3c81" candidate_uuid: "24823002ca7a42eda21a575d14481a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920b396eadfb4a918f207a25d277316f"
I20260620 20:21:59.724296 22461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e389bbd790f4795bc3f2cc4a78d3c81" candidate_uuid: "24823002ca7a42eda21a575d14481a62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "347e99631f9f4497bb50f65d01665ece"
I20260620 20:21:59.724332 22595 raft_consensus.cc:3037] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:59.724416 22461 raft_consensus.cc:3037] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:59.725395 22595 raft_consensus.cc:2454] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24823002ca7a42eda21a575d14481a62 in term 1.
I20260620 20:21:59.725502 22461 raft_consensus.cc:2454] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24823002ca7a42eda21a575d14481a62 in term 1.
I20260620 20:21:59.725740 22261 leader_election.cc:302] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24823002ca7a42eda21a575d14481a62, 920b396eadfb4a918f207a25d277316f; no voters:
I20260620 20:21:59.725898 22664 raft_consensus.cc:2781] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:59.726117 22664 raft_consensus.cc:686] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [term 1 LEADER]: Becoming Leader. State: Replica: 24823002ca7a42eda21a575d14481a62, State: Running, Role: LEADER
I20260620 20:21:59.726419 22664 consensus_queue.cc:237] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } }
I20260620 20:21:59.729301 22182 catalog_manager.cc:5495] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24823002ca7a42eda21a575d14481a62 (127.2.235.193). New cstate: current_term: 1 leader_uuid: "24823002ca7a42eda21a575d14481a62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24823002ca7a42eda21a575d14481a62" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 } health_report { overall_health: UNKNOWN } } }
I20260620 20:21:59.800836 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.797657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.2.235.193:44383)
0620 20:21:59.797787 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.797790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.797804 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:21:59.797933 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.797935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.797988 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.798077 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:21:59.798273 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.798752 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.799567 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.799798 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.799880 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.800119 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:21:59.800509 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:21:59.800600 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.800662 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.800722 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:21:59.806214 22461 raft_consensus.cc:1261] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 1 FOLLOWER]: Refusing update from remote peer 24823002ca7a42eda21a575d14481a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:59.806214 22595 raft_consensus.cc:1261] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 1 FOLLOWER]: Refusing update from remote peer 24823002ca7a42eda21a575d14481a62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:21:59.806859 22664 consensus_queue.cc:1035] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "920b396eadfb4a918f207a25d277316f" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:59.807092 22669 consensus_queue.cc:1035] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 24823002ca7a42eda21a575d14481a62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "347e99631f9f4497bb50f65d01665ece" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:21:59.813040 22674 mvcc.cc:205] Tried to move back new op lower bound from 7299018423519617024 to 7299018423201886208. Current Snapshot: MvccSnapshot[applied={T|T < 7299018423519617024}]
I20260620 20:21:59.813407 22672 mvcc.cc:205] Tried to move back new op lower bound from 7299018423519617024 to 7299018423201886208. Current Snapshot: MvccSnapshot[applied={T|T < 7299018423519617024}]
I20260620 20:21:59.816159 22673 mvcc.cc:205] Tried to move back new op lower bound from 7299018423519617024 to 7299018423201886208. Current Snapshot: MvccSnapshot[applied={T|T < 7299018423519617024}]
I20260620 20:21:59.821285 22181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:59.821435 22181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:21:59.823531 22181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e78dce6edf4815a6c8b4061fa4d017: Sending DeleteTablet for 3 replicas of tablet 3e389bbd790f4795bc3f2cc4a78d3c81
I20260620 20:21:59.824179 22575 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e389bbd790f4795bc3f2cc4a78d3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:59 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:58748
I20260620 20:21:59.824199 22307 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e389bbd790f4795bc3f2cc4a78d3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:59 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:45104
I20260620 20:21:59.824199 22441 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e389bbd790f4795bc3f2cc4a78d3c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:21:59 UTC) from {username='kudu', principal='kudu/127.2.235.254@KRBTEST.COM'} at 127.0.0.1:44028
I20260620 20:21:59.824819 22682 tablet_replica.cc:330] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: stopping tablet replica
I20260620 20:21:59.825080 22682 raft_consensus.cc:2229] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:59.825330 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22244
I20260620 20:21:59.825383 22682 raft_consensus.cc:2258] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:59.825695 22684 tablet_replica.cc:330] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: stopping tablet replica
I20260620 20:21:59.825946 22684 raft_consensus.cc:2229] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:21:59.826196 22684 raft_consensus.cc:2258] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:21:59.827977 22684 ts_tablet_manager.cc:1905] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:59.828425 22682 ts_tablet_manager.cc:1905] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:21:59.830298 22684 ts_tablet_manager.cc:1918] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:59.830406 22684 log.cc:1196] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/3e389bbd790f4795bc3f2cc4a78d3c81
I20260620 20:21:59.830729 22684 ts_tablet_manager.cc:1939] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 347e99631f9f4497bb50f65d01665ece: Deleting consensus metadata
I20260620 20:21:59.831782 22168 catalog_manager.cc:4841] TS 347e99631f9f4497bb50f65d01665ece (127.2.235.194:46787): tablet 3e389bbd790f4795bc3f2cc4a78d3c81 (table test-table [id=e102e3a416e64ff9993f194e72f527fd]) successfully deleted
I20260620 20:21:59.831900 22682 ts_tablet_manager.cc:1918] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:21:59.832005 22682 log.cc:1196] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/3e389bbd790f4795bc3f2cc4a78d3c81
I20260620 20:21:59.832365 22682 ts_tablet_manager.cc:1939] T 3e389bbd790f4795bc3f2cc4a78d3c81 P 920b396eadfb4a918f207a25d277316f: Deleting consensus metadata
W20260620 20:21:59.833583 22168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:44383 (error 108)
I20260620 20:21:59.833636 22167 catalog_manager.cc:4841] TS 920b396eadfb4a918f207a25d277316f (127.2.235.195:39691): tablet 3e389bbd790f4795bc3f2cc4a78d3c81 (table test-table [id=e102e3a416e64ff9993f194e72f527fd]) successfully deleted
I20260620 20:21:59.833854 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22378
I20260620 20:21:59.835543 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.835293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:44383 (local address 127.0.0.1:45136)
0620 20:21:59.835411 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.835482 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:21:59.835717 22168 catalog_manager.cc:4584] TS 24823002ca7a42eda21a575d14481a62 (127.2.235.193:44383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e389bbd790f4795bc3f2cc4a78d3c81: Network error: Client connection negotiation failed: client connection to 127.2.235.193:44383: connect: Connection refused (error 111)
I20260620 20:21:59.840353 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22512
I20260620 20:21:59.845553 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22150
2026-06-20T20:21:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2438 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T20:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:59Z Disabled control of system clock
I20260620 20:21:59.874541 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40873
--webserver_interface=127.2.235.254
--webserver_port=0
--builtin_ntp_servers=127.2.235.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:40873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:21:59.978214 22689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:59.978420 22689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:59.978497 22689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:59.982230 22689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:59.982276 22689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:59.982304 22689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:59.982321 22689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:59.982336 22689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:59.986660 22689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:40873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:21:59.987439 22689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:59.988555 22689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:59.996224 22695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.996261 22694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.996261 22697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:59.996322 22689 server_base.cc:1034] running on GCE node
I20260620 20:21:59.997058 22689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:59.998059 22689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:59.999234 22689 hybrid_clock.cc:648] HybridClock initialized: now 1781986919999208 us; error 39 us; skew 500 ppm
I20260620 20:22:00.000998 22689 webserver.cc:460] Webserver started at http://127.2.235.254:40183/ using document root <none> and password file <none>
I20260620 20:22:00.001684 22689 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:00.001791 22689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:00.002014 22689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:00.003813 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "42b94a55e240415bb30d45e629415a43"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "3b8443111756c36b6f900dd823408619"
server_key_iv: "0cdb253c68da402c1622c27c1b4f716f"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.004329 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "42b94a55e240415bb30d45e629415a43"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "3b8443111756c36b6f900dd823408619"
server_key_iv: "0cdb253c68da402c1622c27c1b4f716f"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.007741 22689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:22:00.010111 22703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.011158 22689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:22:00.011303 22689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "42b94a55e240415bb30d45e629415a43"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "3b8443111756c36b6f900dd823408619"
server_key_iv: "0cdb253c68da402c1622c27c1b4f716f"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.011400 22689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:00.023437 22689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:00.023947 22689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:00.024109 22689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:00.030426 22689 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:40873
I20260620 20:22:00.030433 22755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:40873 every 8 connection(s)
I20260620 20:22:00.031204 22689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:00.033895 22756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:00.038710 22756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43: Bootstrap starting.
I20260620 20:22:00.039480 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22689
I20260620 20:22:00.039606 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:00.039834 2991 external_mini_cluster.cc:1428] Setting key 11ae693b3d7ce94145ba27f2096aac33
I20260620 20:22:00.041081 22756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:00.041802 22756 log.cc:824] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:00.043440 22756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43: No bootstrap required, opened a new log
I20260620 20:22:00.046715 22756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b94a55e240415bb30d45e629415a43" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40873 } }
I20260620 20:22:00.046916 22756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:00.047003 22756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b94a55e240415bb30d45e629415a43, State: Initialized, Role: FOLLOWER
I20260620 20:22:00.047428 22756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b94a55e240415bb30d45e629415a43" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40873 } }
I20260620 20:22:00.047583 22756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:00.047659 22756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:00.047768 22756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:00.048029 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.040726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.2.235.254:40873)
0620 20:22:00.041339 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.041352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.041382 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:00.041958 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.041974 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.042389 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.042715 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.043075 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.043965 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.044826 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.045342 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.045506 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.046382 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.046387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.046397 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:00.046712 (+ 315us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.046767 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:00.046783 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.046879 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.047048 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.047288 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260620 20:22:00.048557 22756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b94a55e240415bb30d45e629415a43" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40873 } }
I20260620 20:22:00.048892 22756 leader_election.cc:302] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42b94a55e240415bb30d45e629415a43; no voters:
I20260620 20:22:00.049160 22756 leader_election.cc:288] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:00.049244 22761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:00.049489 22761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [term 1 LEADER]: Becoming Leader. State: Replica: 42b94a55e240415bb30d45e629415a43, State: Running, Role: LEADER
I20260620 20:22:00.049835 22761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b94a55e240415bb30d45e629415a43" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40873 } }
I20260620 20:22:00.050469 22756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:00.050827 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42b94a55e240415bb30d45e629415a43. Latest consensus state: current_term: 1 leader_uuid: "42b94a55e240415bb30d45e629415a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b94a55e240415bb30d45e629415a43" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40873 } } }
I20260620 20:22:00.050992 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:00.051183 22762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42b94a55e240415bb30d45e629415a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42b94a55e240415bb30d45e629415a43" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 40873 } } }
I20260620 20:22:00.051287 22762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:00.051366 22766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:00.054181 22766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:00.058708 22766 catalog_manager.cc:1347] Generated new cluster ID: d99ea7c06789403c9c51c62a91cdcda1
I20260620 20:22:00.058763 22766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:00.066356 22766 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:00.067085 22766 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:00.075166 22766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43: Generated new TSK 0
I20260620 20:22:00.075824 22766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:00.087917 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.193:0
--local_ip_for_outbound_sockets=127.2.235.193
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40873
--builtin_ntp_servers=127.2.235.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:22:00.191365 22780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:00.191605 22780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:00.191665 22780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:00.196064 22780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:00.196127 22780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:00.196205 22780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.193
I20260620 20:22:00.200433 22780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.235.193
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.193
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:00.201375 22780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:00.202544 22780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:00.211006 22786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.211005 22785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.211004 22788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:00.211222 22780 server_base.cc:1034] running on GCE node
I20260620 20:22:00.211555 22780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:00.212158 22780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:00.213327 22780 hybrid_clock.cc:648] HybridClock initialized: now 1781986920213318 us; error 28 us; skew 500 ppm
I20260620 20:22:00.215137 22780 webserver.cc:460] Webserver started at http://127.2.235.193:42923/ using document root <none> and password file <none>
I20260620 20:22:00.215658 22780 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:00.215754 22780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:00.215975 22780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:00.217804 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data/instance:
uuid: "fe6456d41f8946eab663514c357da6b9"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "23c7ef4b30198edf7bfe9fe488f43f4d"
server_key_iv: "0ca0bfa8ea569e22a65538fa2259a424"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.218323 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance:
uuid: "fe6456d41f8946eab663514c357da6b9"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "23c7ef4b30198edf7bfe9fe488f43f4d"
server_key_iv: "0ca0bfa8ea569e22a65538fa2259a424"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.221432 22780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:00.223443 22794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.224449 22780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:00.224589 22780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal
uuid: "fe6456d41f8946eab663514c357da6b9"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "23c7ef4b30198edf7bfe9fe488f43f4d"
server_key_iv: "0ca0bfa8ea569e22a65538fa2259a424"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.224691 22780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:00.236529 22780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:00.237073 22780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:00.237236 22780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:00.237756 22780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:00.238559 22780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:00.238626 22780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.238690 22780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:00.238745 22780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.246920 22780 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.193:40731
I20260620 20:22:00.246951 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.193:40731 every 8 connection(s)
I20260620 20:22:00.247689 22780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:00.253172 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22780
I20260620 20:22:00.253327 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:00.253597 2991 external_mini_cluster.cc:1428] Setting key 09edc5611a33a4f551d4b5cea2de1567
I20260620 20:22:00.255656 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.194:0
--local_ip_for_outbound_sockets=127.2.235.194
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40873
--builtin_ntp_servers=127.2.235.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:22:00.256583 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.249224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.193:36023 (local address 127.2.235.254:40873)
0620 20:22:00.249393 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.249397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.250268 (+ 871us) server_negotiation.cc:407] Connection header received
0620 20:22:00.251766 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.251770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.251823 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.251948 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.252760 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.253317 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.254554 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.254837 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.254933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.256140 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.256143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.256145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:00.256173 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.256190 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:00.256197 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.256263 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.256441 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.256484 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 20:22:00.257586 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.249562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40873 (local address 127.2.235.193:36023)
0620 20:22:00.250147 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.250178 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.251541 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.252096 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.252102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.252175 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:00.252639 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.252649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.253434 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.253436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.254436 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.254445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.255015 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.255020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.255210 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.255838 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:00.255842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:00.255896 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:00.255984 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:00.256284 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:00.256296 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:00.256305 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.256435 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.256710 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260620 20:22:00.258670 22908 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40873
I20260620 20:22:00.258906 22908 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:00.259343 22908 heartbeater.cc:511] Master 127.2.235.254:40873 requested a full tablet report, sending...
I20260620 20:22:00.260640 22720 ts_manager.cc:194] Registered new tserver with Master: fe6456d41f8946eab663514c357da6b9 (127.2.235.193:40731)
I20260620 20:22:00.261756 22720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.193:36023
W20260620 20:22:00.355755 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:00.355962 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:00.356010 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:00.359617 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:00.359656 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:00.359732 22911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.194
I20260620 20:22:00.363986 22911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.235.194
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.194
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:00.364845 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:00.366058 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:00.375006 22917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.375041 22919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.375041 22916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:00.375347 22911 server_base.cc:1034] running on GCE node
I20260620 20:22:00.375763 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:00.376334 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:00.377502 22911 hybrid_clock.cc:648] HybridClock initialized: now 1781986920377499 us; error 32 us; skew 500 ppm
I20260620 20:22:00.379180 22911 webserver.cc:460] Webserver started at http://127.2.235.194:35741/ using document root <none> and password file <none>
I20260620 20:22:00.379685 22911 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:00.379771 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:00.379987 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:00.381939 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data/instance:
uuid: "0eca56d94d204ca29e5016bc9d8126a2"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "75510b037a2891dc4e151fa29c748465"
server_key_iv: "831e8658eff69a191b1875ae2a2281c5"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.382448 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance:
uuid: "0eca56d94d204ca29e5016bc9d8126a2"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "75510b037a2891dc4e151fa29c748465"
server_key_iv: "831e8658eff69a191b1875ae2a2281c5"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.385641 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:00.387766 22925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.388798 22911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:00.388913 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal
uuid: "0eca56d94d204ca29e5016bc9d8126a2"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "75510b037a2891dc4e151fa29c748465"
server_key_iv: "831e8658eff69a191b1875ae2a2281c5"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.389007 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:00.400203 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:00.400699 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:00.400864 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:00.401423 22911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:00.402205 22911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:00.402271 22911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.402331 22911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:00.402372 22911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.411005 22911 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.194:45915
I20260620 20:22:00.411026 23038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.194:45915 every 8 connection(s)
I20260620 20:22:00.411840 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:00.420133 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.413544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.194:53707 (local address 127.2.235.254:40873)
0620 20:22:00.413671 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.413675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.414704 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 20:22:00.416068 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.416071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.416115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.416187 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.416959 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.417551 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.418627 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.418825 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.418902 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.419761 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.419764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.419766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:00.419776 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.419793 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:00.419800 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.419857 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.419983 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.420029 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:22:00.420814 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.413905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40873 (local address 127.2.235.194:53707)
0620 20:22:00.414561 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.414597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.415894 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.416337 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.416343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.416417 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:00.416845 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.416855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.417660 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.417663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.418518 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.418527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.418943 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.418945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.419065 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.419545 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:00.419547 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:00.419581 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:00.419633 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:00.419869 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:00.419875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:00.419879 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.419962 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.420143 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
I20260620 20:22:00.421680 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 22911
I20260620 20:22:00.421773 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:00.421859 23039 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40873
I20260620 20:22:00.422071 23039 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:00.422079 2991 external_mini_cluster.cc:1428] Setting key 5f7b21295002bbf6643f3588b65eae4f
I20260620 20:22:00.422549 23039 heartbeater.cc:511] Master 127.2.235.254:40873 requested a full tablet report, sending...
I20260620 20:22:00.423483 22720 ts_manager.cc:194] Registered new tserver with Master: 0eca56d94d204ca29e5016bc9d8126a2 (127.2.235.194:45915)
I20260620 20:22:00.423847 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.235.195:0
--local_ip_for_outbound_sockets=127.2.235.195
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40873
--builtin_ntp_servers=127.2.235.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:22:00.424121 22720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.194:53707
W20260620 20:22:00.525207 23042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:00.525429 23042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:00.525462 23042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:00.529450 23042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:00.529495 23042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:00.529618 23042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.235.195
I20260620 20:22:00.534150 23042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.235.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.235.195
--webserver_port=0
--tserver_master_addrs=127.2.235.254:40873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.235.195
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:00.534978 23042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:00.536170 23042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:00.544414 23047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:00.544449 23042 server_base.cc:1034] running on GCE node
W20260620 20:22:00.544437 23048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.544431 23050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:00.545074 23042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:00.545676 23042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:00.546859 23042 hybrid_clock.cc:648] HybridClock initialized: now 1781986920546835 us; error 44 us; skew 500 ppm
I20260620 20:22:00.548684 23042 webserver.cc:460] Webserver started at http://127.2.235.195:41005/ using document root <none> and password file <none>
I20260620 20:22:00.549226 23042 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:00.549357 23042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:00.549561 23042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:00.551476 23042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data/instance:
uuid: "49199422a1ce49ca987844fc8b1298a8"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "3d595a24e01b9b52ea23c8e78e77688d"
server_key_iv: "db2cf930ba9db588be0c5e0cf7eaa3ce"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.552073 23042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance:
uuid: "49199422a1ce49ca987844fc8b1298a8"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "3d595a24e01b9b52ea23c8e78e77688d"
server_key_iv: "db2cf930ba9db588be0c5e0cf7eaa3ce"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.555325 23042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:22:00.557451 23056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.558467 23042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:00.558593 23042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal
uuid: "49199422a1ce49ca987844fc8b1298a8"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-8z53"
server_key: "3d595a24e01b9b52ea23c8e78e77688d"
server_key_iv: "db2cf930ba9db588be0c5e0cf7eaa3ce"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.558692 23042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:00.578025 23042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:00.578570 23042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:00.578740 23042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:00.579285 23042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:00.580101 23042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:00.580170 23042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.580236 23042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:00.580292 23042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.588634 23042 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.195:44767
I20260620 20:22:00.588665 23169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.195:44767 every 8 connection(s)
I20260620 20:22:00.589516 23042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:00.598477 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.591122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:39269 (local address 127.2.235.254:40873)
0620 20:22:00.591237 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.591241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.592148 (+ 907us) server_negotiation.cc:407] Connection header received
0620 20:22:00.593571 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.593574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.593632 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.593704 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.594488 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.595063 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.596420 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.596671 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.596811 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.597933 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.597937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.597940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:00.597954 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.597986 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:00.597995 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.598081 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.598260 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.598324 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:22:00.599501 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.591436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:40873 (local address 127.2.235.195:39269)
0620 20:22:00.592018 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.592054 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.593382 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.593855 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.593861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.593938 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:00.594347 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.594358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.595185 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.595190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.596275 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.596286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.596843 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.596847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.597017 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.597670 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:00.597671 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:00.597707 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:00.597771 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:00.598082 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:00.598092 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:00.598101 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.598234 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.598480 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260620 20:22:00.599638 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23042
I20260620 20:22:00.599778 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:00.600092 2991 external_mini_cluster.cc:1428] Setting key 1773700eca31b178c009e2cda45d42a7
I20260620 20:22:00.600565 23170 heartbeater.cc:347] Connected to a master server at 127.2.235.254:40873
I20260620 20:22:00.600762 23170 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:00.601233 23170 heartbeater.cc:511] Master 127.2.235.254:40873 requested a full tablet report, sending...
I20260620 20:22:00.602303 22720 ts_manager.cc:194] Registered new tserver with Master: 49199422a1ce49ca987844fc8b1298a8 (127.2.235.195:44767)
I20260620 20:22:00.602805 22720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.235.195:39269
I20260620 20:22:00.613462 2991 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:22:00.618423 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.615384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.2.235.254:40873)
0620 20:22:00.615547 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.615551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.615745 (+ 194us) server_negotiation.cc:407] Connection header received
0620 20:22:00.615841 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.615843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.615881 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.615937 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.616125 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.616533 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.617220 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.617441 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.617507 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.618091 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.618094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.618096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:00.618105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.618120 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:00.618128 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.618181 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.618231 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.618270 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:22:00.620586 22720 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:00.623255 22720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:00.636526 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.631843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.2.235.193:40731)
0620 20:22:00.632167 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.632179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.632205 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:22:00.632282 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.632288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.632469 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.632652 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:00.632929 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.633698 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.635301 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.635871 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.636007 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.636195 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.636277 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.636359 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":106,"threads_started":1}
I20260620 20:22:00.636700 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.631051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:45915 (local address 127.0.0.1:48114)
0620 20:22:00.631561 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.631574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.631659 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.632664 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.632667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.632682 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:00.632765 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.632773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.634226 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.634232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.635230 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.635271 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.636283 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.636286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.636363 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.636384 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.636452 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.636516 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":26,"thread_start_us":75,"threads_started":1}
I20260620 20:22:00.636899 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.630821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.195:44767 (local address 127.0.0.1:55758)
0620 20:22:00.631070 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.631110 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.631212 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.632603 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.632608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.632628 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:00.632755 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.632762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.634067 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.634070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.635197 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.635207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.636486 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.636489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.636582 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.636611 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.636707 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.636781 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":1,"thread_start_us":55,"threads_started":1}
I20260620 20:22:00.636552 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.631465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40731 (local address 127.0.0.1:60010)
0620 20:22:00.631861 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.631874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.631954 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.632738 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.632741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.632754 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:00.632841 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.632848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.633813 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.633817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.635166 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.635177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.636006 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.636010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.636115 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.636147 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.636255 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.636367 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":31,"thread_start_us":78,"threads_started":1}
I20260620 20:22:00.637327 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.630953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.2.235.195:44767)
0620 20:22:00.632191 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.632200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.632218 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:22:00.632279 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.632283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.632424 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.632598 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:00.632849 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.633965 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.635338 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.636140 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.636240 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.636431 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.637055 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.637154 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":145,"threads_started":1}
I20260620 20:22:00.637370 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.631624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.2.235.194:45915)
0620 20:22:00.632269 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.632279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.632298 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:22:00.632361 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.632364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.632516 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.632909 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:00.633047 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.634090 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.635357 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.636140 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.636902 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.637079 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.637119 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.637194 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":106,"threads_started":1}
I20260620 20:22:00.639598 22973 tablet_service.cc:1467] Processing CreateTablet for tablet 436e5da2d8dd4d3d85426cae1ad14d4e (DEFAULT_TABLE table=test-table [id=d271f5c41c594899b7aea905f60f45dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:00.639737 22842 tablet_service.cc:1467] Processing CreateTablet for tablet 436e5da2d8dd4d3d85426cae1ad14d4e (DEFAULT_TABLE table=test-table [id=d271f5c41c594899b7aea905f60f45dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:00.639839 23104 tablet_service.cc:1467] Processing CreateTablet for tablet 436e5da2d8dd4d3d85426cae1ad14d4e (DEFAULT_TABLE table=test-table [id=d271f5c41c594899b7aea905f60f45dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:00.640513 22973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436e5da2d8dd4d3d85426cae1ad14d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:00.640633 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436e5da2d8dd4d3d85426cae1ad14d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:00.640633 22842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436e5da2d8dd4d3d85426cae1ad14d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:00.646056 23187 tablet_bootstrap.cc:492] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Bootstrap starting.
I20260620 20:22:00.647423 23189 tablet_bootstrap.cc:492] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Bootstrap starting.
I20260620 20:22:00.648619 23187 tablet_bootstrap.cc:654] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:00.648988 23189 tablet_bootstrap.cc:654] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:00.649410 23187 log.cc:824] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:00.649537 23188 tablet_bootstrap.cc:492] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: Bootstrap starting.
I20260620 20:22:00.649762 23189 log.cc:824] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:00.651324 23188 tablet_bootstrap.cc:654] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:00.651403 23187 tablet_bootstrap.cc:492] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: No bootstrap required, opened a new log
I20260620 20:22:00.651573 23187 ts_tablet_manager.cc:1397] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 20:22:00.651665 23189 tablet_bootstrap.cc:492] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: No bootstrap required, opened a new log
I20260620 20:22:00.651818 23189 ts_tablet_manager.cc:1397] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:22:00.652020 23188 log.cc:824] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:00.653867 23188 tablet_bootstrap.cc:492] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: No bootstrap required, opened a new log
I20260620 20:22:00.654050 23188 ts_tablet_manager.cc:1397] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 20:22:00.655023 23189 raft_consensus.cc:348] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.655236 23189 raft_consensus.cc:374] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:00.655305 23189 raft_consensus.cc:729] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49199422a1ce49ca987844fc8b1298a8, State: Initialized, Role: FOLLOWER
I20260620 20:22:00.655733 23189 consensus_queue.cc:260] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.655740 23187 raft_consensus.cc:348] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.655990 23187 raft_consensus.cc:374] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:00.656059 23187 raft_consensus.cc:729] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eca56d94d204ca29e5016bc9d8126a2, State: Initialized, Role: FOLLOWER
I20260620 20:22:00.656426 23187 consensus_queue.cc:260] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.656610 23189 ts_tablet_manager.cc:1428] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:22:00.656900 23170 heartbeater.cc:503] Master 127.2.235.254:40873 was elected leader, sending a full tablet report...
I20260620 20:22:00.656944 23039 heartbeater.cc:503] Master 127.2.235.254:40873 was elected leader, sending a full tablet report...
I20260620 20:22:00.657280 23187 ts_tablet_manager.cc:1428] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260620 20:22:00.657845 23188 raft_consensus.cc:348] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.658092 23188 raft_consensus.cc:374] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:00.658145 23188 raft_consensus.cc:729] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6456d41f8946eab663514c357da6b9, State: Initialized, Role: FOLLOWER
I20260620 20:22:00.658521 23188 consensus_queue.cc:260] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.659024 22908 heartbeater.cc:503] Master 127.2.235.254:40873 was elected leader, sending a full tablet report...
I20260620 20:22:00.659314 23188 ts_tablet_manager.cc:1428] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
W20260620 20:22:00.662698 23040 tablet.cc:2369] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:00.748639 22909 tablet.cc:2369] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:00.840402 23171 tablet.cc:2369] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:00.939584 23193 raft_consensus.cc:482] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:00.939810 23193 raft_consensus.cc:504] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.940974 23193 leader_election.cc:288] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6456d41f8946eab663514c357da6b9 (127.2.235.193:40731), 0eca56d94d204ca29e5016bc9d8126a2 (127.2.235.194:45915)
I20260620 20:22:00.944401 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.941210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40731 (local address 127.2.235.195:38267)
0620 20:22:00.941335 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.941350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.941436 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.941843 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.941845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.941862 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:00.941967 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.941971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.942636 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.942639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.943355 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.943363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.943974 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.943977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.944068 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.944098 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.944200 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.944270 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:22:00.944402 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.941447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:38267 (local address 127.2.235.193:40731)
0620 20:22:00.941603 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.941606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.941617 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:22:00.941676 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.941678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.941716 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.941816 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:00.942069 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.942533 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.943470 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.943858 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.943982 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.944030 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.944216 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.944288 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:22:00.944813 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.941460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.235.195:33571 (local address 127.2.235.194:45915)
0620 20:22:00.941576 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.941582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.941812 (+ 230us) server_negotiation.cc:407] Connection header received
0620 20:22:00.941967 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.941971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.942022 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.942105 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:00.942384 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.942830 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.943827 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.944232 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.944368 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.944417 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.944616 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.944688 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:22:00.944834 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.941210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.194:45915 (local address 127.2.235.195:33571)
0620 20:22:00.941691 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.941703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.941855 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.942115 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.942118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.942133 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:00.942272 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.942277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.942938 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.942944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.943692 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.943700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.944366 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.944371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.944476 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.944510 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.944596 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.944655 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":139,"thread_start_us":112,"threads_started":1}
I20260620 20:22:00.945062 22862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436e5da2d8dd4d3d85426cae1ad14d4e" candidate_uuid: "49199422a1ce49ca987844fc8b1298a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6456d41f8946eab663514c357da6b9" is_pre_election: true
I20260620 20:22:00.945411 22862 raft_consensus.cc:2454] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49199422a1ce49ca987844fc8b1298a8 in term 0.
I20260620 20:22:00.945463 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436e5da2d8dd4d3d85426cae1ad14d4e" candidate_uuid: "49199422a1ce49ca987844fc8b1298a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eca56d94d204ca29e5016bc9d8126a2" is_pre_election: true
I20260620 20:22:00.945786 22993 raft_consensus.cc:2454] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49199422a1ce49ca987844fc8b1298a8 in term 0.
I20260620 20:22:00.945963 23058 leader_election.cc:302] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49199422a1ce49ca987844fc8b1298a8, fe6456d41f8946eab663514c357da6b9; no voters:
I20260620 20:22:00.946276 23193 raft_consensus.cc:2781] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:00.946362 23193 raft_consensus.cc:482] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:00.946409 23193 raft_consensus.cc:3037] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:00.947397 23193 raft_consensus.cc:504] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.947731 23193 leader_election.cc:288] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [CANDIDATE]: Term 1 election: Requested vote from peers fe6456d41f8946eab663514c357da6b9 (127.2.235.193:40731), 0eca56d94d204ca29e5016bc9d8126a2 (127.2.235.194:45915)
I20260620 20:22:00.948032 22862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436e5da2d8dd4d3d85426cae1ad14d4e" candidate_uuid: "49199422a1ce49ca987844fc8b1298a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6456d41f8946eab663514c357da6b9"
I20260620 20:22:00.948099 22993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436e5da2d8dd4d3d85426cae1ad14d4e" candidate_uuid: "49199422a1ce49ca987844fc8b1298a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eca56d94d204ca29e5016bc9d8126a2"
I20260620 20:22:00.948143 22862 raft_consensus.cc:3037] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:00.948222 22993 raft_consensus.cc:3037] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:00.949152 22862 raft_consensus.cc:2454] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49199422a1ce49ca987844fc8b1298a8 in term 1.
I20260620 20:22:00.949157 22993 raft_consensus.cc:2454] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49199422a1ce49ca987844fc8b1298a8 in term 1.
I20260620 20:22:00.949537 23058 leader_election.cc:302] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49199422a1ce49ca987844fc8b1298a8, fe6456d41f8946eab663514c357da6b9; no voters:
I20260620 20:22:00.949787 23193 raft_consensus.cc:2781] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:00.949986 23193 raft_consensus.cc:686] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 1 LEADER]: Becoming Leader. State: Replica: 49199422a1ce49ca987844fc8b1298a8, State: Running, Role: LEADER
I20260620 20:22:00.950649 23193 consensus_queue.cc:237] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } }
I20260620 20:22:00.954020 22720 catalog_manager.cc:5495] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49199422a1ce49ca987844fc8b1298a8 (127.2.235.195). New cstate: current_term: 1 leader_uuid: "49199422a1ce49ca987844fc8b1298a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49199422a1ce49ca987844fc8b1298a8" member_type: VOTER last_known_addr { host: "127.2.235.195" port: 44767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:00.993127 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.989868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.2.235.195:44767)
0620 20:22:00.989976 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.989979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.990018 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:00.990142 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.990144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.990185 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.990259 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:00.990446 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.990878 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.991698 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.991903 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.992031 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.992346 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:00.992706 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:00.992863 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.992926 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.992976 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:22:00.998673 22993 raft_consensus.cc:1261] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 1 FOLLOWER]: Refusing update from remote peer 49199422a1ce49ca987844fc8b1298a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:00.998674 22862 raft_consensus.cc:1261] T 436e5da2d8dd4d3d85426cae1ad14d4e P fe6456d41f8946eab663514c357da6b9 [term 1 FOLLOWER]: Refusing update from remote peer 49199422a1ce49ca987844fc8b1298a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:00.999325 23193 consensus_queue.cc:1035] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6456d41f8946eab663514c357da6b9" member_type: VOTER last_known_addr { host: "127.2.235.193" port: 40731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:00.999500 23197 consensus_queue.cc:1035] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eca56d94d204ca29e5016bc9d8126a2" member_type: VOTER last_known_addr { host: "127.2.235.194" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:01.007097 23201 mvcc.cc:205] Tried to move back new op lower bound from 7299018428403273728 to 7299018428217131008. Current Snapshot: MvccSnapshot[applied={T|T < 7299018428403273728}]
I20260620 20:22:01.013535 22718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:01.013689 22718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:01.016229 22718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42b94a55e240415bb30d45e629415a43: Sending DeleteTablet for 3 replicas of tablet 436e5da2d8dd4d3d85426cae1ad14d4e
I20260620 20:22:01.016944 22973 tablet_service.cc:1514] Processing DeleteTablet for tablet 436e5da2d8dd4d3d85426cae1ad14d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:01 UTC) from {username='slave'} at 127.0.0.1:48114
I20260620 20:22:01.016945 22842 tablet_service.cc:1514] Processing DeleteTablet for tablet 436e5da2d8dd4d3d85426cae1ad14d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:01 UTC) from {username='slave'} at 127.0.0.1:60010
I20260620 20:22:01.016973 23104 tablet_service.cc:1514] Processing DeleteTablet for tablet 436e5da2d8dd4d3d85426cae1ad14d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:01 UTC) from {username='slave'} at 127.0.0.1:55758
I20260620 20:22:01.017436 23211 tablet_replica.cc:330] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: stopping tablet replica
I20260620 20:22:01.017653 23210 tablet_replica.cc:330] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: stopping tablet replica
I20260620 20:22:01.017920 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22780
I20260620 20:22:01.019609 23211 raft_consensus.cc:2229] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:01.019925 23211 raft_consensus.cc:2258] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:01.020687 23210 raft_consensus.cc:2229] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:01.020977 23210 raft_consensus.cc:2258] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:01.021461 23211 ts_tablet_manager.cc:1905] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:01.022169 23210 ts_tablet_manager.cc:1905] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:01.024677 23210 ts_tablet_manager.cc:1918] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:01.024878 23210 log.cc:1196] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-1/wal/wals/436e5da2d8dd4d3d85426cae1ad14d4e
I20260620 20:22:01.024895 23211 ts_tablet_manager.cc:1918] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:01.025008 23211 log.cc:1196] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Deleting WAL directory at /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986686246774-2991-0/minicluster-data/ts-2/wal/wals/436e5da2d8dd4d3d85426cae1ad14d4e
I20260620 20:22:01.025385 23210 ts_tablet_manager.cc:1939] T 436e5da2d8dd4d3d85426cae1ad14d4e P 0eca56d94d204ca29e5016bc9d8126a2: Deleting consensus metadata
I20260620 20:22:01.025393 23211 ts_tablet_manager.cc:1939] T 436e5da2d8dd4d3d85426cae1ad14d4e P 49199422a1ce49ca987844fc8b1298a8: Deleting consensus metadata
I20260620 20:22:01.026554 22705 catalog_manager.cc:4841] TS 49199422a1ce49ca987844fc8b1298a8 (127.2.235.195:44767): tablet 436e5da2d8dd4d3d85426cae1ad14d4e (table test-table [id=d271f5c41c594899b7aea905f60f45dc]) successfully deleted
I20260620 20:22:01.026870 22707 catalog_manager.cc:4841] TS 0eca56d94d204ca29e5016bc9d8126a2 (127.2.235.194:45915): tablet 436e5da2d8dd4d3d85426cae1ad14d4e (table test-table [id=d271f5c41c594899b7aea905f60f45dc]) successfully deleted
W20260620 20:22:01.026993 22705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.235.193:40731 (error 108)
I20260620 20:22:01.027747 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22911
I20260620 20:22:01.027971 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:01.027649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.193:40731 (local address 127.0.0.1:60022)
0620 20:22:01.027800 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:01.027880 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 20:22:01.028816 22705 catalog_manager.cc:4584] TS fe6456d41f8946eab663514c357da6b9 (127.2.235.193:40731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 436e5da2d8dd4d3d85426cae1ad14d4e: Network error: Client connection negotiation failed: client connection to 127.2.235.193:40731: connect: Connection refused (error 111)
I20260620 20:22:01.035162 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23042
I20260620 20:22:01.043623 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 22689
2026-06-20T20:22:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1196 ms)
[----------] 6 tests from EncryptionPolicyTest (10047 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:01 dist-test-slave-8z53 krb5kdc[23215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:01 dist-test-slave-8z53 krb5kdc[23215](info): set up 2 sockets
Jun 20 20:22:01 dist-test-slave-8z53 krb5kdc[23215](info): commencing operation
krb5kdc: starting...
W20260620 20:22:02.013095 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:02 dist-test-slave-8z53 krb5kdc[23215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:02.121832 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:02.219081 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:02.219318 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:02.219388 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:02.223997 23228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:02.224066 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:02.224103 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:02.224131 23228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:02.224160 23228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:02.231222 23228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:02.231971 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:02.232786 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:02.240468 23233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:02.240408 23234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:02.240617 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:02.240648 23228 server_base.cc:1034] running on GCE node
I20260620 20:22:02.241088 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:02.242050 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:02.243201 23228 hybrid_clock.cc:648] HybridClock initialized: now 1781986922243176 us; error 37 us; skew 500 ppm
Jun 20 20:22:02 dist-test-slave-8z53 krb5kdc[23215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:02.248368 23228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:02.249187 23228 webserver.cc:460] Webserver started at http://127.0.0.1:46043/ using document root <none> and password file <none>
I20260620 20:22:02.249683 23228 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:02.249739 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:02.250002 23228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:02.251827 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "e03f06dff2044a638c7ec1ff74b74838"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-8z53"
server_key: "33ab6831d3fb2e7d7dcf313f547b6924"
server_key_iv: "253c569731c60e0b64ded570172fecc4"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.252334 23228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "e03f06dff2044a638c7ec1ff74b74838"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-8z53"
server_key: "33ab6831d3fb2e7d7dcf313f547b6924"
server_key_iv: "253c569731c60e0b64ded570172fecc4"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.255687 23228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:22:02.257850 23243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:02.258894 23228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:22:02.259006 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "e03f06dff2044a638c7ec1ff74b74838"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-8z53"
server_key: "33ab6831d3fb2e7d7dcf313f547b6924"
server_key_iv: "253c569731c60e0b64ded570172fecc4"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.259095 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:02.284677 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:02.287812 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:02.288007 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:02.294539 23228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42069
I20260620 20:22:02.294561 23295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42069 every 8 connection(s)
I20260620 20:22:02.295375 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:02.296603 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23228
I20260620 20:22:02.296736 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:02.296988 2991 external_mini_cluster.cc:1428] Setting key 1981421bf9d1045757e51b157e51430e
I20260620 20:22:02.298220 23296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:02.304229 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838: Bootstrap starting.
Jun 20 20:22:02 dist-test-slave-8z53 krb5kdc[23215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:22:02.306438 23296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:02.307344 23296 log.cc:824] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:02.309301 23296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838: No bootstrap required, opened a new log
I20260620 20:22:02.311209 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:02.298304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35493 (local address 127.0.0.1:42069)
0620 20:22:02.298706 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:02.298722 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:02.298759 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:22:02.299492 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:02.299515 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:02.299876 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:02.300187 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:02.300781 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:02.302146 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:02.302935 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:02.303583 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:02.303742 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:02.306231 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:02.306236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:02.306259 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:02.306292 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:02.308055 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:02.308507 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:02.308512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:02.308518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:02.308643 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:02.308949 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:02.308952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:02.308954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:02.309422 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:02.309617 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:02.309863 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 20:22:02.310050 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260620 20:22:02.312831 23296 raft_consensus.cc:348] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f06dff2044a638c7ec1ff74b74838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260620 20:22:02.313056 23296 raft_consensus.cc:374] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:02.313127 23296 raft_consensus.cc:729] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03f06dff2044a638c7ec1ff74b74838, State: Initialized, Role: FOLLOWER
I20260620 20:22:02.313527 23296 consensus_queue.cc:260] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f06dff2044a638c7ec1ff74b74838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260620 20:22:02.313660 23296 raft_consensus.cc:388] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:02.313721 23296 raft_consensus.cc:482] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:02.313862 23296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:02.313894 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:02.314638 23296 raft_consensus.cc:504] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f06dff2044a638c7ec1ff74b74838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260620 20:22:02.314939 23296 leader_election.cc:302] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e03f06dff2044a638c7ec1ff74b74838; no voters:
I20260620 20:22:02.315284 23296 leader_election.cc:288] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:02.315379 23305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:02.315629 23305 raft_consensus.cc:686] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [term 1 LEADER]: Becoming Leader. State: Replica: e03f06dff2044a638c7ec1ff74b74838, State: Running, Role: LEADER
I20260620 20:22:02.315995 23305 consensus_queue.cc:237] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f06dff2044a638c7ec1ff74b74838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } }
I20260620 20:22:02.316521 23296 sys_catalog.cc:564] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:02.320025 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e03f06dff2044a638c7ec1ff74b74838. Latest consensus state: current_term: 1 leader_uuid: "e03f06dff2044a638c7ec1ff74b74838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f06dff2044a638c7ec1ff74b74838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } } }
I20260620 20:22:02.320176 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:02.320461 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e03f06dff2044a638c7ec1ff74b74838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03f06dff2044a638c7ec1ff74b74838" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42069 } } }
I20260620 20:22:02.320564 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838 [sys.catalog]: This master's current role is: LEADER
W20260620 20:22:02.321666 23321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:02.321774 23321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:02.321838 23323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:02.322593 23323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:02.324749 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:02.315905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38891 (local address 127.0.0.1:42069)
0620 20:22:02.316002 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:02.316006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:02.316299 (+ 293us) server_negotiation.cc:407] Connection header received
0620 20:22:02.316492 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:02.316495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:02.316540 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:02.316619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:02.317578 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:02.318304 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:02.319463 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:02.319860 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:02.319971 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:02.321441 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:02.321445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:02.321447 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:02.321482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:02.323023 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:02.323519 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:02.323522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:02.323523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:02.323595 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:02.323915 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:02.323919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:02.323920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:02.324107 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:02.324246 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:02.324531 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 20:22:02.324615 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:22:02.327638 23323 catalog_manager.cc:1347] Generated new cluster ID: 3650ae7210114db4b3ac191b8f78743d
I20260620 20:22:02.327692 23323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:02.353837 23323 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:02.354898 23323 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:02.362119 23323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e03f06dff2044a638c7ec1ff74b74838: Generated new TSK 0
I20260620 20:22:02.362807 23323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:02.370728 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23228
W20260620 20:22:02.376156 23213 connection.cc:535] client connection to 127.0.0.1:42069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42069): Connection reset by peer (error 104)
2026-06-20T20:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:02 dist-test-slave-8z53 krb5kdc[23327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:02 dist-test-slave-8z53 krb5kdc[23327](info): set up 2 sockets
Jun 20 20:22:02 dist-test-slave-8z53 krb5kdc[23327](info): commencing operation
krb5kdc: starting...
W20260620 20:22:03.341303 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:03 dist-test-slave-8z53 krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:03.445747 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:03.543931 23340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:03.544160 23340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:03.544234 23340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:03.548044 23340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:03.548111 23340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:03.548144 23340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:03.548192 23340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:03.548224 23340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:03.552567 23340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:03.553438 23340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:03.554307 23340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:03.562233 23348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:03.562227 23346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:03.562336 23345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:03.562418 23340 server_base.cc:1034] running on GCE node
I20260620 20:22:03.563143 23340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:03.564055 23340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:03.565300 23340 hybrid_clock.cc:648] HybridClock initialized: now 1781986923565265 us; error 42 us; skew 500 ppm
Jun 20 20:22:03 dist-test-slave-8z53 krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:03.567850 23340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:03.568647 23340 webserver.cc:460] Webserver started at http://127.0.0.1:41787/ using document root <none> and password file <none>
I20260620 20:22:03.569176 23340 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:03.569299 23340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:03.569528 23340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:03.571492 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "0c290e73262a4dc7bd63356bbed6644b"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-8z53"
server_key: "da1bc2d5df48fc9353d258d6ca81c216"
server_key_iv: "45f1e7ace8dbbcdae8ce958fdc5813e2"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.572042 23340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "0c290e73262a4dc7bd63356bbed6644b"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-8z53"
server_key: "da1bc2d5df48fc9353d258d6ca81c216"
server_key_iv: "45f1e7ace8dbbcdae8ce958fdc5813e2"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.575527 23340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:22:03.577752 23355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.578863 23340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:22:03.579005 23340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "0c290e73262a4dc7bd63356bbed6644b"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-8z53"
server_key: "da1bc2d5df48fc9353d258d6ca81c216"
server_key_iv: "45f1e7ace8dbbcdae8ce958fdc5813e2"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.579108 23340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:03.594177 23340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:03.596621 23340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:03.596800 23340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:03.603766 23340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34745
I20260620 20:22:03.603806 23407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34745 every 8 connection(s)
I20260620 20:22:03.604671 23340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:03.607211 23408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:03.610354 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23340
I20260620 20:22:03.610509 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:03.610833 2991 external_mini_cluster.cc:1428] Setting key f031e8fff562d6b979f872fce0abe83c
I20260620 20:22:03.612605 23408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b: Bootstrap starting.
I20260620 20:22:03.614646 23408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:03.615324 23408 log.cc:824] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:03.617072 23408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b: No bootstrap required, opened a new log
I20260620 20:22:03.619732 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:03.612140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39669 (local address 127.0.0.1:34745)
0620 20:22:03.612536 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:03.612550 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:03.612594 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:22:03.613318 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:03.613342 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:03.613666 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:03.614013 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:03.614421 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.615552 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.616482 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.617029 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.617217 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:03.618148 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:03.618153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:03.618164 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:03.618517 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:03.618569 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:03.618592 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:03.618747 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:03.618935 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 20:22:03.619084 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20260620 20:22:03.620610 23408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c290e73262a4dc7bd63356bbed6644b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34745 } }
I20260620 20:22:03.620877 23408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:03.620923 23408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c290e73262a4dc7bd63356bbed6644b, State: Initialized, Role: FOLLOWER
I20260620 20:22:03.621342 23408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c290e73262a4dc7bd63356bbed6644b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34745 } }
I20260620 20:22:03.621522 23408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:03.621588 23408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:03.621683 23408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:03.622582 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:03.622605 23408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c290e73262a4dc7bd63356bbed6644b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34745 } }
I20260620 20:22:03.622920 23408 leader_election.cc:302] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c290e73262a4dc7bd63356bbed6644b; no voters:
I20260620 20:22:03.623193 23408 leader_election.cc:288] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:03.623276 23413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:03.623432 23413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [term 1 LEADER]: Becoming Leader. State: Replica: 0c290e73262a4dc7bd63356bbed6644b, State: Running, Role: LEADER
I20260620 20:22:03.623750 23413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c290e73262a4dc7bd63356bbed6644b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34745 } }
I20260620 20:22:03.624310 23408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:03.625356 23414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c290e73262a4dc7bd63356bbed6644b. Latest consensus state: current_term: 1 leader_uuid: "0c290e73262a4dc7bd63356bbed6644b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c290e73262a4dc7bd63356bbed6644b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34745 } } }
I20260620 20:22:03.625483 23414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:03.625746 23415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c290e73262a4dc7bd63356bbed6644b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c290e73262a4dc7bd63356bbed6644b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34745 } } }
I20260620 20:22:03.625818 23415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:03.626173 23427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:03.629179 23427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:03.630264 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:03.624745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54267 (local address 127.0.0.1:34745)
0620 20:22:03.625045 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:03.625050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:03.625322 (+ 272us) server_negotiation.cc:407] Connection header received
0620 20:22:03.625519 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:03.625523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:03.625568 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:03.625625 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:03.626197 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.627101 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.628293 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.628665 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.628789 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:03.629707 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:03.629711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:03.629714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:03.629727 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:03.629756 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:03.629773 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:03.629905 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:03.630065 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:22:03.630100 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260620 20:22:03.634501 23427 catalog_manager.cc:1347] Generated new cluster ID: ca9061ce649d4aafbbd88f93182e85f2
I20260620 20:22:03.634559 23427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:03.641007 23427 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:03.641860 23427 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:03.649291 23427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c290e73262a4dc7bd63356bbed6644b: Generated new TSK 0
I20260620 20:22:03.649904 23427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:03.653801 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23340
W20260620 20:22:03.658660 23325 connection.cc:535] client connection to 127.0.0.1:34745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34745): Connection reset by peer (error 104)
2026-06-20T20:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:03 dist-test-slave-8z53 krb5kdc[23439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:03 dist-test-slave-8z53 krb5kdc[23439](info): set up 2 sockets
Jun 20 20:22:03 dist-test-slave-8z53 krb5kdc[23439](info): commencing operation
krb5kdc: starting...
W20260620 20:22:04.613564 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:04 dist-test-slave-8z53 krb5kdc[23439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:04.717553 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:04.829630 23452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:04.829876 23452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:04.829953 23452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:04.833748 23452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:04.833817 23452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:04.833853 23452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:04.833904 23452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:04.833940 23452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:04.838657 23452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:04.839466 23452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:04.840304 23452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:04.847734 23460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:04.847831 23457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:04.847872 23452 server_base.cc:1034] running on GCE node
W20260620 20:22:04.847735 23458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:04.848469 23452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:04.849406 23452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:04.850549 23452 hybrid_clock.cc:648] HybridClock initialized: now 1781986924850527 us; error 35 us; skew 500 ppm
Jun 20 20:22:04 dist-test-slave-8z53 krb5kdc[23439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:04.852957 23452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:04.853706 23452 webserver.cc:460] Webserver started at http://127.0.0.1:37265/ using document root <none> and password file <none>
I20260620 20:22:04.854159 23452 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:04.854219 23452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:04.854401 23452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:04.856159 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "1bed1f32ba6a47f686d27f473fce347f"
format_stamp: "Formatted at 2026-06-20 20:22:04 on dist-test-slave-8z53"
server_key: "c9aa9f59253870d964140fc7b0059d5b"
server_key_iv: "3e2e53acd84f760d6ab4fd6b508dffc7"
server_key_version: "encryptionkey@0"
I20260620 20:22:04.856683 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "1bed1f32ba6a47f686d27f473fce347f"
format_stamp: "Formatted at 2026-06-20 20:22:04 on dist-test-slave-8z53"
server_key: "c9aa9f59253870d964140fc7b0059d5b"
server_key_iv: "3e2e53acd84f760d6ab4fd6b508dffc7"
server_key_version: "encryptionkey@0"
I20260620 20:22:04.859732 23452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:22:04.861901 23467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:04.862900 23452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:04.863006 23452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "1bed1f32ba6a47f686d27f473fce347f"
format_stamp: "Formatted at 2026-06-20 20:22:04 on dist-test-slave-8z53"
server_key: "c9aa9f59253870d964140fc7b0059d5b"
server_key_iv: "3e2e53acd84f760d6ab4fd6b508dffc7"
server_key_version: "encryptionkey@0"
I20260620 20:22:04.863094 23452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:04.867036 23452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:04.869445 23452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:04.869601 23452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:04.875679 23452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36595
I20260620 20:22:04.875764 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36595 every 8 connection(s)
I20260620 20:22:04.876495 23452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:04.878940 23520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:04.881834 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23452
I20260620 20:22:04.881985 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:04.882251 2991 external_mini_cluster.cc:1428] Setting key e380b5730f125af34e3e25ed9a2fb771
I20260620 20:22:04.884217 23520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f: Bootstrap starting.
I20260620 20:22:04.886763 23520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:04.887238 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.883460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41719 (local address 127.0.0.1:36595)
0620 20:22:04.883861 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.883876 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.883915 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:04.884709 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.884731 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.885043 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.885440 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.886023 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.886028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.886040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:04.886050 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.886102 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:04.886128 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.886229 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.886347 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.886484 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":98,"threads_started":1}
I20260620 20:22:04.887545 23520 log.cc:824] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:04.889384 23520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f: No bootstrap required, opened a new log
I20260620 20:22:04.889739 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:04.892944 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.891632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37817 (local address 127.0.0.1:36595)
0620 20:22:04.891752 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.891756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.891855 (+ 99us) server_negotiation.cc:407] Connection header received
0620 20:22:04.891939 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.891944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.891980 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.892080 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.892622 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.892625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.892628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:04.892638 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.892661 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:04.892672 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.892759 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.892801 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.892831 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:22:04.893106 23520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bed1f32ba6a47f686d27f473fce347f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36595 } }
I20260620 20:22:04.893352 23520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:04.893416 23520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bed1f32ba6a47f686d27f473fce347f, State: Initialized, Role: FOLLOWER
I20260620 20:22:04.893818 23520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bed1f32ba6a47f686d27f473fce347f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36595 } }
I20260620 20:22:04.893931 23520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:04.893973 23520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:04.894032 23520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:04.894680 23520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bed1f32ba6a47f686d27f473fce347f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36595 } }
I20260620 20:22:04.894930 23520 leader_election.cc:302] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bed1f32ba6a47f686d27f473fce347f; no voters:
I20260620 20:22:04.895140 23520 leader_election.cc:288] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:04.895241 23530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:04.895496 23530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [term 1 LEADER]: Becoming Leader. State: Replica: 1bed1f32ba6a47f686d27f473fce347f, State: Running, Role: LEADER
I20260620 20:22:04.895807 23530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bed1f32ba6a47f686d27f473fce347f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36595 } }
I20260620 20:22:04.896273 23520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:04.897338 23531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bed1f32ba6a47f686d27f473fce347f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bed1f32ba6a47f686d27f473fce347f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36595 } } }
I20260620 20:22:04.897459 23531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:04.897765 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bed1f32ba6a47f686d27f473fce347f. Latest consensus state: current_term: 1 leader_uuid: "1bed1f32ba6a47f686d27f473fce347f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bed1f32ba6a47f686d27f473fce347f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36595 } } }
I20260620 20:22:04.897856 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:04.898175 23539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:04.900738 23539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:04.905542 23539 catalog_manager.cc:1347] Generated new cluster ID: 051b9f40edd4440db2cc2e2aa6eb2857
I20260620 20:22:04.905599 23539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:04.934743 23539 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:04.935016 23539 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:04.950217 23539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bed1f32ba6a47f686d27f473fce347f: Generated new TSK 0
I20260620 20:22:04.950887 23539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:04.954516 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23452
2026-06-20T20:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:05 dist-test-slave-8z53 krb5kdc[23551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:05 dist-test-slave-8z53 krb5kdc[23551](info): set up 2 sockets
Jun 20 20:22:05 dist-test-slave-8z53 krb5kdc[23551](info): commencing operation
krb5kdc: starting...
W20260620 20:22:05.921263 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:05 dist-test-slave-8z53 krb5kdc[23551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:06.025166 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:06.122256 23564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:06.122447 23564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:06.122483 23564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:06.125999 23564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:06.126044 23564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:06.126063 23564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:06.126080 23564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:06.126093 23564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:06.130708 23564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:06.131408 23564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:06.132148 23564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:06.139415 23569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:06.139423 23572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:06.139423 23570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:06.139612 23564 server_base.cc:1034] running on GCE node
I20260620 20:22:06.140117 23564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:06.140974 23564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:06.142122 23564 hybrid_clock.cc:648] HybridClock initialized: now 1781986926142107 us; error 24 us; skew 500 ppm
Jun 20 20:22:06 dist-test-slave-8z53 krb5kdc[23551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:06.144550 23564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:06.145382 23564 webserver.cc:460] Webserver started at http://127.0.0.1:39491/ using document root <none> and password file <none>
I20260620 20:22:06.145893 23564 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:06.145984 23564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:06.146200 23564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:06.148036 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "27e421d1cafd40fd98992b126ef5886f"
format_stamp: "Formatted at 2026-06-20 20:22:06 on dist-test-slave-8z53"
server_key: "fedfee08e9a545c323a4a0871898a5db"
server_key_iv: "9f91e985f108e8b183ca93799c6f4950"
server_key_version: "encryptionkey@0"
I20260620 20:22:06.148561 23564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "27e421d1cafd40fd98992b126ef5886f"
format_stamp: "Formatted at 2026-06-20 20:22:06 on dist-test-slave-8z53"
server_key: "fedfee08e9a545c323a4a0871898a5db"
server_key_iv: "9f91e985f108e8b183ca93799c6f4950"
server_key_version: "encryptionkey@0"
I20260620 20:22:06.151984 23564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 20:22:06.154073 23579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:06.155004 23564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:06.155131 23564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "27e421d1cafd40fd98992b126ef5886f"
format_stamp: "Formatted at 2026-06-20 20:22:06 on dist-test-slave-8z53"
server_key: "fedfee08e9a545c323a4a0871898a5db"
server_key_iv: "9f91e985f108e8b183ca93799c6f4950"
server_key_version: "encryptionkey@0"
I20260620 20:22:06.155225 23564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:06.166658 23564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:06.168989 23564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:06.169153 23564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:06.175169 23564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40685
I20260620 20:22:06.175179 23631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40685 every 8 connection(s)
I20260620 20:22:06.175977 23564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:06.178323 23632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:06.179584 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23564
I20260620 20:22:06.179719 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:06.179994 2991 external_mini_cluster.cc:1428] Setting key d4f5c422c38f6fe9098e8aad32b28ff1
I20260620 20:22:06.184204 23632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f: Bootstrap starting.
I20260620 20:22:06.186465 23632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:06.187215 23632 log.cc:824] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:22:06 dist-test-slave-8z53 krb5kdc[23551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:22:06.188843 23632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f: No bootstrap required, opened a new log
I20260620 20:22:06.191828 23632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e421d1cafd40fd98992b126ef5886f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 20:22:06.192070 23632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:06.192155 23632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e421d1cafd40fd98992b126ef5886f, State: Initialized, Role: FOLLOWER
I20260620 20:22:06.192559 23632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e421d1cafd40fd98992b126ef5886f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 20:22:06.192696 23632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:06.192791 23632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:06.192791 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:06.181101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37045 (local address 127.0.0.1:40685)
0620 20:22:06.181475 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:06.181492 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:06.181528 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:22:06.182243 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:06.182261 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:06.182622 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:06.182939 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:06.183512 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:06.184775 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:06.185633 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:06.186248 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:06.186445 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:06.188686 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:06.188691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:06.188702 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:06.188724 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:06.190256 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:06.190673 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:06.190679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:06.190684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:06.190783 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:06.191049 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:06.191052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:06.191053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:06.191373 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:06.191558 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:06.191796 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:22:06.192043 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":138,"threads_started":1}
I20260620 20:22:06.192894 23632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:06.193801 23632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e421d1cafd40fd98992b126ef5886f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 20:22:06.194085 23632 leader_election.cc:302] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e421d1cafd40fd98992b126ef5886f; no voters:
I20260620 20:22:06.194392 23632 leader_election.cc:288] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:06.194454 23637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:06.194701 23637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [term 1 LEADER]: Becoming Leader. State: Replica: 27e421d1cafd40fd98992b126ef5886f, State: Running, Role: LEADER
I20260620 20:22:06.195029 23637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e421d1cafd40fd98992b126ef5886f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 20:22:06.195338 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:06.195564 23632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:06.196467 23639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e421d1cafd40fd98992b126ef5886f. Latest consensus state: current_term: 1 leader_uuid: "27e421d1cafd40fd98992b126ef5886f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e421d1cafd40fd98992b126ef5886f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } } }
I20260620 20:22:06.196609 23639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:06.196938 23638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e421d1cafd40fd98992b126ef5886f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e421d1cafd40fd98992b126ef5886f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } } }
I20260620 20:22:06.197022 23638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:06.198037 23648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:06.200896 23648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:06.205430 23635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:06.197593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50949 (local address 127.0.0.1:40685)
0620 20:22:06.197672 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:06.197675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:06.197964 (+ 289us) server_negotiation.cc:407] Connection header received
0620 20:22:06.198107 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:06.198110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:06.198150 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:06.198192 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:06.198840 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:06.199542 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:06.200606 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:06.200884 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:06.201011 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:06.202386 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:06.202389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:06.202392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:06.202412 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:06.203833 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:06.204242 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:06.204244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:06.204246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:06.204317 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:06.204587 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:06.204592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:06.204597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:06.204765 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:06.204886 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:06.205120 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:22:06.205220 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 20:22:06.206389 23648 catalog_manager.cc:1347] Generated new cluster ID: 66fc5c636fd941a38852d08311f9467d
I20260620 20:22:06.206473 23648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:06.217315 23648 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:06.218020 23648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:06.223902 23648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27e421d1cafd40fd98992b126ef5886f: Generated new TSK 0
I20260620 20:22:06.224396 23648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:06.234390 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23564
2026-06-20T20:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:06 dist-test-slave-8z53 krb5kdc[23663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:06 dist-test-slave-8z53 krb5kdc[23663](info): set up 2 sockets
Jun 20 20:22:06 dist-test-slave-8z53 krb5kdc[23663](info): commencing operation
krb5kdc: starting...
W20260620 20:22:07.205092 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:07 dist-test-slave-8z53 krb5kdc[23663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:07.309963 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:07.412539 23676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:07.412935 23676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:07.412972 23676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:07.416678 23676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:07.416741 23676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:07.416759 23676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:07.416774 23676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:07.416787 23676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:07.421869 23676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:07.422636 23676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:07.423444 23676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:07.431483 23684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:07.431469 23682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:07.431500 23681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:07.431867 23676 server_base.cc:1034] running on GCE node
I20260620 20:22:07.432376 23676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:07.433357 23676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:07.434538 23676 hybrid_clock.cc:648] HybridClock initialized: now 1781986927434515 us; error 38 us; skew 500 ppm
Jun 20 20:22:07 dist-test-slave-8z53 krb5kdc[23663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:07.441638 23676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:07.442513 23676 webserver.cc:460] Webserver started at http://127.0.0.1:46183/ using document root <none> and password file <none>
I20260620 20:22:07.443032 23676 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:07.443118 23676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:07.443362 23676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:07.445188 23676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "3bea5ea7b81b4973b257a732af93e639"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-8z53"
server_key: "9ccbb55b1190beab9ef11fbd1ffcdf6e"
server_key_iv: "ce0af11c703b480dabdc8598925d5eac"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.445816 23676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "3bea5ea7b81b4973b257a732af93e639"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-8z53"
server_key: "9ccbb55b1190beab9ef11fbd1ffcdf6e"
server_key_iv: "ce0af11c703b480dabdc8598925d5eac"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.449179 23676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:07.451411 23691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:07.452616 23676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:22:07.452747 23676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "3bea5ea7b81b4973b257a732af93e639"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-8z53"
server_key: "9ccbb55b1190beab9ef11fbd1ffcdf6e"
server_key_iv: "ce0af11c703b480dabdc8598925d5eac"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.452867 23676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:07.465108 23676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:07.467955 23676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:07.468111 23676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:07.474366 23676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34603
I20260620 20:22:07.474385 23743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34603 every 8 connection(s)
I20260620 20:22:07.475152 23676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:07.477753 23744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:07.482983 23744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639: Bootstrap starting.
I20260620 20:22:07.484478 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23676
I20260620 20:22:07.484668 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:07.484932 2991 external_mini_cluster.cc:1428] Setting key b6e19f713bba9481b4db359735d6f544
I20260620 20:22:07.485757 23744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:07.486627 23744 log.cc:824] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:07.488597 23744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639: No bootstrap required, opened a new log
I20260620 20:22:07.492100 23744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bea5ea7b81b4973b257a732af93e639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34603 } }
I20260620 20:22:07.492312 23744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:07.492393 23744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bea5ea7b81b4973b257a732af93e639, State: Initialized, Role: FOLLOWER
I20260620 20:22:07.492816 23744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bea5ea7b81b4973b257a732af93e639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34603 } }
I20260620 20:22:07.492941 23744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:07.493036 23744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:07.493146 23744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:07.494125 23744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bea5ea7b81b4973b257a732af93e639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34603 } }
I20260620 20:22:07.494385 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.486007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48799 (local address 127.0.0.1:34603)
0620 20:22:07.486461 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:07.486482 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:07.486526 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:22:07.487289 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:07.487312 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:07.487679 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:07.488004 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:07.488411 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.489732 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.490572 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.491769 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.491925 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.492709 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:07.492714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:07.492729 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:07.493159 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:07.493223 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:07.493276 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:07.493362 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:07.493514 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 20:22:07.493694 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260620 20:22:07.494426 23744 leader_election.cc:302] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bea5ea7b81b4973b257a732af93e639; no voters:
I20260620 20:22:07.494794 23744 leader_election.cc:288] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:07.494812 23749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:07.495002 23749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [term 1 LEADER]: Becoming Leader. State: Replica: 3bea5ea7b81b4973b257a732af93e639, State: Running, Role: LEADER
I20260620 20:22:07.496071 23744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:07.496203 23749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bea5ea7b81b4973b257a732af93e639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34603 } }
I20260620 20:22:07.497045 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:07.498958 23751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bea5ea7b81b4973b257a732af93e639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bea5ea7b81b4973b257a732af93e639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34603 } } }
I20260620 20:22:07.499074 23751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:07.499390 23750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bea5ea7b81b4973b257a732af93e639. Latest consensus state: current_term: 1 leader_uuid: "3bea5ea7b81b4973b257a732af93e639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bea5ea7b81b4973b257a732af93e639" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34603 } } }
I20260620 20:22:07.499630 23750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:07.499650 23768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:07.502424 23768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:07.502743 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.498850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39131 (local address 127.0.0.1:34603)
0620 20:22:07.498963 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:07.498967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:07.499203 (+ 236us) server_negotiation.cc:407] Connection header received
0620 20:22:07.499374 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:07.499376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:07.499413 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:07.499459 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:07.499828 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.500263 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.501200 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.501529 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.501634 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.502318 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:07.502321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:07.502324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:07.502334 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:07.502356 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:07.502370 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:07.502433 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:07.502554 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:22:07.502617 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:22:07.508773 23768 catalog_manager.cc:1347] Generated new cluster ID: cf518c264bcd44e5a81bd67271f9635c
I20260620 20:22:07.508829 23768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:07.514245 23768 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:07.515115 23768 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:07.524313 23768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bea5ea7b81b4973b257a732af93e639: Generated new TSK 0
I20260620 20:22:07.525002 23768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:07.531255 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23676
2026-06-20T20:22:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:07 dist-test-slave-8z53 krb5kdc[23775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:07 dist-test-slave-8z53 krb5kdc[23775](info): set up 2 sockets
Jun 20 20:22:07 dist-test-slave-8z53 krb5kdc[23775](info): commencing operation
krb5kdc: starting...
W20260620 20:22:08.517381 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:08 dist-test-slave-8z53 krb5kdc[23775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:08.623342 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:08.724298 23788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:08.724529 23788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:08.724565 23788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:08.728549 23788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:08.728595 23788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:08.728621 23788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:08.728654 23788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:08.728690 23788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:08.733639 23788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:08.734478 23788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:08.735333 23788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:22:08.742942 23788 server_base.cc:1034] running on GCE node
W20260620 20:22:08.742897 23793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:08.742936 23796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:08.742898 23794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:08.743655 23788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:08.744544 23788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:08.745697 23788 hybrid_clock.cc:648] HybridClock initialized: now 1781986928745678 us; error 29 us; skew 500 ppm
Jun 20 20:22:08 dist-test-slave-8z53 krb5kdc[23775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:08.748160 23788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:08.748873 23788 webserver.cc:460] Webserver started at http://127.0.0.1:35747/ using document root <none> and password file <none>
I20260620 20:22:08.749377 23788 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:08.749439 23788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:08.749584 23788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:08.751277 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "d5c7e00940674b75a48d4f315b65f091"
format_stamp: "Formatted at 2026-06-20 20:22:08 on dist-test-slave-8z53"
server_key: "a7abd4f30da4009e239a95a8672943b2"
server_key_iv: "5cf2ac44e2b10a98b871b5b8df5dea5e"
server_key_version: "encryptionkey@0"
I20260620 20:22:08.751726 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "d5c7e00940674b75a48d4f315b65f091"
format_stamp: "Formatted at 2026-06-20 20:22:08 on dist-test-slave-8z53"
server_key: "a7abd4f30da4009e239a95a8672943b2"
server_key_iv: "5cf2ac44e2b10a98b871b5b8df5dea5e"
server_key_version: "encryptionkey@0"
I20260620 20:22:08.755002 23788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:22:08.757135 23803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:08.758217 23788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:08.758330 23788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "d5c7e00940674b75a48d4f315b65f091"
format_stamp: "Formatted at 2026-06-20 20:22:08 on dist-test-slave-8z53"
server_key: "a7abd4f30da4009e239a95a8672943b2"
server_key_iv: "5cf2ac44e2b10a98b871b5b8df5dea5e"
server_key_version: "encryptionkey@0"
I20260620 20:22:08.758419 23788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:08.762363 23788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:08.765549 23788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:08.765753 23788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:08.771917 23788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44471
I20260620 20:22:08.771965 23855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44471 every 8 connection(s)
I20260620 20:22:08.772683 23788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:08.775102 23856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.777930 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23788
I20260620 20:22:08.778056 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:08.778347 2991 external_mini_cluster.cc:1428] Setting key 8d81fed9278e2ab409b0bf824d036998
I20260620 20:22:08.780375 23856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091: Bootstrap starting.
I20260620 20:22:08.782734 23856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.783133 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.779542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36139 (local address 127.0.0.1:44471)
0620 20:22:08.779907 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.779923 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.779962 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:08.780679 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.780699 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.780991 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.781331 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:08.781914 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:08.781919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:08.781932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:08.781941 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:08.781989 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:08.782009 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:08.782124 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.782198 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.782336 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20260620 20:22:08.783516 23856 log.cc:824] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:08.785327 23856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091: No bootstrap required, opened a new log
I20260620 20:22:08.785532 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:08.788828 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.787369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51253 (local address 127.0.0.1:44471)
0620 20:22:08.787464 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.787467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.787695 (+ 228us) server_negotiation.cc:407] Connection header received
0620 20:22:08.787805 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.787807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.787830 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.787894 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:08.788528 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:08.788530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:08.788531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:08.788539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:08.788556 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:08.788564 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:08.788617 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.788728 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.788753 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:22:08.789549 23856 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c7e00940674b75a48d4f315b65f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260620 20:22:08.789795 23856 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.789862 23856 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c7e00940674b75a48d4f315b65f091, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.790290 23856 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c7e00940674b75a48d4f315b65f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260620 20:22:08.790398 23856 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:08.790465 23856 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:08.790563 23856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.791297 23856 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c7e00940674b75a48d4f315b65f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260620 20:22:08.791610 23856 leader_election.cc:302] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c7e00940674b75a48d4f315b65f091; no voters:
I20260620 20:22:08.791936 23856 leader_election.cc:288] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:08.792059 23866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:08.792280 23866 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [term 1 LEADER]: Becoming Leader. State: Replica: d5c7e00940674b75a48d4f315b65f091, State: Running, Role: LEADER
I20260620 20:22:08.792613 23866 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c7e00940674b75a48d4f315b65f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260620 20:22:08.793079 23856 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:08.794342 23868 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c7e00940674b75a48d4f315b65f091. Latest consensus state: current_term: 1 leader_uuid: "d5c7e00940674b75a48d4f315b65f091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c7e00940674b75a48d4f315b65f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } } }
I20260620 20:22:08.794507 23868 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:08.794341 23867 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c7e00940674b75a48d4f315b65f091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c7e00940674b75a48d4f315b65f091" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } } }
I20260620 20:22:08.794780 23867 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091 [sys.catalog]: This master's current role is: LEADER
W20260620 20:22:08.797636 23881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:08.797713 23881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:08.797775 23877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:08.798401 23877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:08.802785 23877 catalog_manager.cc:1347] Generated new cluster ID: 584e738015a04f25ad8ae92f359df5a3
I20260620 20:22:08.802841 23877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:08.824442 23877 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:08.824700 23877 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:08.831478 23877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c7e00940674b75a48d4f315b65f091: Generated new TSK 0
I20260620 20:22:08.832031 23877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:08.841617 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23788
2026-06-20T20:22:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:08 dist-test-slave-8z53 krb5kdc[23887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:08 dist-test-slave-8z53 krb5kdc[23887](info): set up 2 sockets
Jun 20 20:22:08 dist-test-slave-8z53 krb5kdc[23887](info): commencing operation
krb5kdc: starting...
W20260620 20:22:09.829598 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:09 dist-test-slave-8z53 krb5kdc[23887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:09.936249 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:10.036311 23900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:10.036520 23900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:10.036558 23900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:10.040380 23900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:10.040422 23900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:10.040439 23900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:10.040454 23900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:10.040496 23900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:10.045015 23900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:10.045910 23900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:10.046793 23900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:10.054296 23905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:10.054322 23908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:10.054322 23900 server_base.cc:1034] running on GCE node
W20260620 20:22:10.054482 23906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:10.055011 23900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:10.055976 23900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:10.057150 23900 hybrid_clock.cc:648] HybridClock initialized: now 1781986930057141 us; error 29 us; skew 500 ppm
Jun 20 20:22:10 dist-test-slave-8z53 krb5kdc[23887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:10.059520 23900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:10.060201 23900 webserver.cc:460] Webserver started at http://127.0.0.1:45487/ using document root <none> and password file <none>
I20260620 20:22:10.060623 23900 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:10.060675 23900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:10.060810 23900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:10.062505 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "306afc12f72d405f9d0edfedefb01cf3"
format_stamp: "Formatted at 2026-06-20 20:22:10 on dist-test-slave-8z53"
server_key: "591916795966333ff602f3967bf9195a"
server_key_iv: "2947e430cb112866a8a92a0524e1dde4"
server_key_version: "encryptionkey@0"
I20260620 20:22:10.062932 23900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "306afc12f72d405f9d0edfedefb01cf3"
format_stamp: "Formatted at 2026-06-20 20:22:10 on dist-test-slave-8z53"
server_key: "591916795966333ff602f3967bf9195a"
server_key_iv: "2947e430cb112866a8a92a0524e1dde4"
server_key_version: "encryptionkey@0"
I20260620 20:22:10.066064 23900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:10.068301 23915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:10.069419 23900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:22:10.069515 23900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "306afc12f72d405f9d0edfedefb01cf3"
format_stamp: "Formatted at 2026-06-20 20:22:10 on dist-test-slave-8z53"
server_key: "591916795966333ff602f3967bf9195a"
server_key_iv: "2947e430cb112866a8a92a0524e1dde4"
server_key_version: "encryptionkey@0"
I20260620 20:22:10.069597 23900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:10.086055 23900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:10.088474 23900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:10.088608 23900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:10.095304 23900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34217
I20260620 20:22:10.095335 23967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34217 every 8 connection(s)
I20260620 20:22:10.096184 23900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:10.098629 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:10.101037 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 23900
I20260620 20:22:10.101176 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:10.101486 2991 external_mini_cluster.cc:1428] Setting key 73333c53734c1915dc28d9bc51d33370
I20260620 20:22:10.103726 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3: Bootstrap starting.
I20260620 20:22:10.105741 23968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:10.106439 23968 log.cc:824] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:10.108143 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3: No bootstrap required, opened a new log
Jun 20 20:22:10 dist-test-slave-8z53 krb5kdc[23887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:22:10.111582 23968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306afc12f72d405f9d0edfedefb01cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260620 20:22:10.111792 23968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:10.111863 23968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306afc12f72d405f9d0edfedefb01cf3, State: Initialized, Role: FOLLOWER
I20260620 20:22:10.112267 23968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306afc12f72d405f9d0edfedefb01cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260620 20:22:10.112401 23968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:10.112466 23968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:10.112571 23968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:10.113431 23968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306afc12f72d405f9d0edfedefb01cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260620 20:22:10.113770 23968 leader_election.cc:302] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 306afc12f72d405f9d0edfedefb01cf3; no voters:
I20260620 20:22:10.114063 23968 leader_election.cc:288] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:10.114125 23973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:10.114387 23973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [term 1 LEADER]: Becoming Leader. State: Replica: 306afc12f72d405f9d0edfedefb01cf3, State: Running, Role: LEADER
I20260620 20:22:10.114701 23973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306afc12f72d405f9d0edfedefb01cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260620 20:22:10.115152 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:10.102856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:41237 (local address 127.0.0.1:34217)
0620 20:22:10.103240 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:10.103252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:10.103282 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:10.103794 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:10.103808 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:10.104132 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:10.104432 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:10.105146 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:10.106182 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:10.107077 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:10.107538 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:10.107687 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:10.110046 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:10.110051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:10.110062 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:10.110085 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:10.111906 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:10.112451 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:10.112457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:10.112464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:10.112631 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:10.112991 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:10.112994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:10.112996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:10.113482 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:10.113701 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:10.114038 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 20:22:10.114289 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
I20260620 20:22:10.115293 23968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:10.116374 23975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "306afc12f72d405f9d0edfedefb01cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306afc12f72d405f9d0edfedefb01cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } } }
I20260620 20:22:10.116667 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 306afc12f72d405f9d0edfedefb01cf3. Latest consensus state: current_term: 1 leader_uuid: "306afc12f72d405f9d0edfedefb01cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "306afc12f72d405f9d0edfedefb01cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } } }
I20260620 20:22:10.116777 23975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:10.116876 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:10.117388 23984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:10.118322 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:10.120831 23984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:10.126436 23984 catalog_manager.cc:1347] Generated new cluster ID: e25c8ac7f50642fc92e6c8cef2a480af
I20260620 20:22:10.126492 23984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:10.132345 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:10.121435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:39903 (local address 127.0.0.1:34217)
0620 20:22:10.121546 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:10.121550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:10.121824 (+ 274us) server_negotiation.cc:407] Connection header received
0620 20:22:10.121954 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:10.121957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:10.122013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:10.122089 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:10.122862 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:10.123335 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:10.124135 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:10.124366 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:10.124472 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:10.126060 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:10.126064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:10.126067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:10.126095 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:10.130550 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:10.131082 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:10.131085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:10.131087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:10.131170 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:10.131517 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:10.131520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:10.131522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:10.131719 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:10.131851 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:10.132110 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:22:10.132194 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:22:10.153730 23984 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:10.154457 23984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:10.160521 23984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 306afc12f72d405f9d0edfedefb01cf3: Generated new TSK 0
I20260620 20:22:10.161051 23984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:10.165063 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 23900
W20260620 20:22:10.170341 23885 connection.cc:535] client connection to 127.0.0.1:34217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34217): Connection reset by peer (error 104)
2026-06-20T20:22:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:10 dist-test-slave-8z53 krb5kdc[23999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:10 dist-test-slave-8z53 krb5kdc[23999](info): set up 2 sockets
Jun 20 20:22:10 dist-test-slave-8z53 krb5kdc[23999](info): commencing operation
krb5kdc: starting...
W20260620 20:22:11.137432 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:11 dist-test-slave-8z53 krb5kdc[23999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:11.241864 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:11.340960 24012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:11.341182 24012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:11.341217 24012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:11.345055 24012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:11.345103 24012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:11.345120 24012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:11.345136 24012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:11.345150 24012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:11.349901 24012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:11.350699 24012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:11.351516 24012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:11.359272 24017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:11.359293 24020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:11.359342 24012 server_base.cc:1034] running on GCE node
W20260620 20:22:11.359297 24018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:11.360052 24012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:11.360991 24012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:11.362160 24012 hybrid_clock.cc:648] HybridClock initialized: now 1781986931362123 us; error 53 us; skew 500 ppm
Jun 20 20:22:11 dist-test-slave-8z53 krb5kdc[23999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:11.364599 24012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:11.365540 24012 webserver.cc:460] Webserver started at http://127.0.0.1:33121/ using document root <none> and password file <none>
I20260620 20:22:11.366040 24012 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:11.366119 24012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:11.366320 24012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:11.368125 24012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "a41a1b51a35241a6bf09246f39301902"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-8z53"
server_key: "c2f90fcc575b4d54c1e41a3501c15d73"
server_key_iv: "39409265ea9224c032e7d80d83e2226b"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.368657 24012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "a41a1b51a35241a6bf09246f39301902"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-8z53"
server_key: "c2f90fcc575b4d54c1e41a3501c15d73"
server_key_iv: "39409265ea9224c032e7d80d83e2226b"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.372036 24012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:22:11.374220 24027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:11.375257 24012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:22:11.375376 24012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "a41a1b51a35241a6bf09246f39301902"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-8z53"
server_key: "c2f90fcc575b4d54c1e41a3501c15d73"
server_key_iv: "39409265ea9224c032e7d80d83e2226b"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.375476 24012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:11.390034 24012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:11.392439 24012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:11.392617 24012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:11.398883 24012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43791
I20260620 20:22:11.398896 24079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43791 every 8 connection(s)
I20260620 20:22:11.399688 24012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:11.402099 24080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:11.406270 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24012
I20260620 20:22:11.406412 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:11.406718 2991 external_mini_cluster.cc:1428] Setting key e8d325e67d71677eebce301f2beb7759
I20260620 20:22:11.407256 24080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902: Bootstrap starting.
I20260620 20:22:11.409376 24080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:11.410109 24080 log.cc:824] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:11.411993 24080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902: No bootstrap required, opened a new log
I20260620 20:22:11.414746 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.407894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:45395 (local address 127.0.0.1:43791)
0620 20:22:11.408244 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:11.408265 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:11.408295 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:11.408853 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:11.408867 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:11.409158 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:11.409441 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:11.409759 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.410696 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.411801 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.412311 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.412430 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.413235 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:11.413240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:11.413273 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:11.413608 (+ 335us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:11.413653 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:11.413674 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:11.413766 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:11.413974 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 20:22:11.414112 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
I20260620 20:22:11.415709 24080 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41a1b51a35241a6bf09246f39301902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260620 20:22:11.415980 24080 raft_consensus.cc:374] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:11.416034 24080 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a1b51a35241a6bf09246f39301902, State: Initialized, Role: FOLLOWER
I20260620 20:22:11.416425 24080 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41a1b51a35241a6bf09246f39301902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260620 20:22:11.416605 24080 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:11.416679 24080 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:11.416796 24080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:11.417480 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:11.417793 24080 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41a1b51a35241a6bf09246f39301902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260620 20:22:11.418135 24080 leader_election.cc:302] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41a1b51a35241a6bf09246f39301902; no voters:
I20260620 20:22:11.418452 24080 leader_election.cc:288] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:11.418566 24085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:11.418758 24085 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [term 1 LEADER]: Becoming Leader. State: Replica: a41a1b51a35241a6bf09246f39301902, State: Running, Role: LEADER
I20260620 20:22:11.419050 24085 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41a1b51a35241a6bf09246f39301902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260620 20:22:11.419639 24080 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:11.420200 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a41a1b51a35241a6bf09246f39301902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41a1b51a35241a6bf09246f39301902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } } }
I20260620 20:22:11.420250 24088 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41a1b51a35241a6bf09246f39301902. Latest consensus state: current_term: 1 leader_uuid: "a41a1b51a35241a6bf09246f39301902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41a1b51a35241a6bf09246f39301902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } } }
I20260620 20:22:11.420365 24088 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:11.420321 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:11.420854 24096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:11.424325 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.419488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:57915 (local address 127.0.0.1:43791)
0620 20:22:11.419574 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:11.419578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:11.420100 (+ 522us) server_negotiation.cc:407] Connection header received
0620 20:22:11.420553 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:11.420556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:11.420606 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:11.420660 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:11.421056 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.421854 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.422649 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.422952 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.423050 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.423827 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:11.423830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:11.423833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:11.423845 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:11.423869 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:11.423880 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:11.423965 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:11.424179 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 20:22:11.424210 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:22:11.424376 24096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:11.429471 24096 catalog_manager.cc:1347] Generated new cluster ID: fc62ec14ba8e4de9ac7f63a0f898c565
I20260620 20:22:11.429530 24096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:11.439050 24096 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:11.440140 24096 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:11.452234 24096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a41a1b51a35241a6bf09246f39301902: Generated new TSK 0
I20260620 20:22:11.452888 24096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:11.459733 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24012
W20260620 20:22:11.464653 23997 connection.cc:535] client connection to 127.0.0.1:43791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43791): Connection reset by peer (error 104)
2026-06-20T20:22:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:11 dist-test-slave-8z53 krb5kdc[24111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:11 dist-test-slave-8z53 krb5kdc[24111](info): set up 2 sockets
Jun 20 20:22:11 dist-test-slave-8z53 krb5kdc[24111](info): commencing operation
krb5kdc: starting...
W20260620 20:22:12.449478 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:12 dist-test-slave-8z53 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:12.552125 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:12.653532 24124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:12.653792 24124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:12.653857 24124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:12.657712 24124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:12.657783 24124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:12.657817 24124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:12.657860 24124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:12.657892 24124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:12.662786 24124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:12.663770 24124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:12.664690 24124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:12.672600 24130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:12.672745 24124 server_base.cc:1034] running on GCE node
W20260620 20:22:12.672652 24132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:12.672608 24129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:12.673415 24124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:12.674356 24124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:12.675554 24124 hybrid_clock.cc:648] HybridClock initialized: now 1781986932675510 us; error 38 us; skew 500 ppm
Jun 20 20:22:12 dist-test-slave-8z53 krb5kdc[24111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:12.681455 24124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:12.682245 24124 webserver.cc:460] Webserver started at http://127.0.0.1:43645/ using document root <none> and password file <none>
I20260620 20:22:12.682723 24124 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:12.682811 24124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:12.683015 24124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:12.684880 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "55897f8ef7f1482e9a381fff75600240"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-8z53"
server_key: "57cb6d89cd48b5f046ec552c2148197d"
server_key_iv: "cea395d496c76d0269666b58b1848501"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.685493 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "55897f8ef7f1482e9a381fff75600240"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-8z53"
server_key: "57cb6d89cd48b5f046ec552c2148197d"
server_key_iv: "cea395d496c76d0269666b58b1848501"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.688967 24124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:22:12.691228 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:12.692355 24124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:22:12.692487 24124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "55897f8ef7f1482e9a381fff75600240"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-8z53"
server_key: "57cb6d89cd48b5f046ec552c2148197d"
server_key_iv: "cea395d496c76d0269666b58b1848501"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.692589 24124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:12.696640 24124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:12.699304 24124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:12.699481 24124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:12.705803 24124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37249
I20260620 20:22:12.705817 24191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37249 every 8 connection(s)
I20260620 20:22:12.706637 24124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:12.706888 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24124
I20260620 20:22:12.707008 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:12.707353 2991 external_mini_cluster.cc:1428] Setting key 7de147a3e7629fda6cc67f060b623357
I20260620 20:22:12.709476 24192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:12.711736 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.708523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:36261 (local address 127.0.0.1:37249)
0620 20:22:12.708866 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:12.708881 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:12.708918 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:22:12.709573 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:12.709595 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:12.709829 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:12.710063 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:12.710744 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:12.710748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:12.710758 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:12.710766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:12.710810 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:12.710826 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:12.710914 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:12.711011 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 20:22:12.711145 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":119,"threads_started":1}
I20260620 20:22:12.714178 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:12.715080 24192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240: Bootstrap starting.
I20260620 20:22:12.717662 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.716212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:54169 (local address 127.0.0.1:37249)
0620 20:22:12.716325 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:12.716330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:12.716480 (+ 150us) server_negotiation.cc:407] Connection header received
0620 20:22:12.716568 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:12.716571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:12.716600 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:12.716676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:12.717323 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:12.717325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:12.717328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:12.717338 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:12.717361 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:12.717371 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:12.717449 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:12.717499 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 20:22:12.717533 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:22:12.717665 24192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:12.718470 24192 log.cc:824] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:12.720060 24192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240: No bootstrap required, opened a new log
I20260620 20:22:12.723486 24192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55897f8ef7f1482e9a381fff75600240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260620 20:22:12.723670 24192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:12.723742 24192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55897f8ef7f1482e9a381fff75600240, State: Initialized, Role: FOLLOWER
I20260620 20:22:12.724103 24192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55897f8ef7f1482e9a381fff75600240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260620 20:22:12.724232 24192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:12.724323 24192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:12.724435 24192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:12.725136 24192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55897f8ef7f1482e9a381fff75600240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260620 20:22:12.725445 24192 leader_election.cc:302] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55897f8ef7f1482e9a381fff75600240; no voters:
I20260620 20:22:12.725692 24192 leader_election.cc:288] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:12.725785 24202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:12.726029 24202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [term 1 LEADER]: Becoming Leader. State: Replica: 55897f8ef7f1482e9a381fff75600240, State: Running, Role: LEADER
I20260620 20:22:12.726390 24202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55897f8ef7f1482e9a381fff75600240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } }
I20260620 20:22:12.726822 24192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:12.727896 24203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55897f8ef7f1482e9a381fff75600240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55897f8ef7f1482e9a381fff75600240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260620 20:22:12.728030 24203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:12.728386 24211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:12.728367 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55897f8ef7f1482e9a381fff75600240. Latest consensus state: current_term: 1 leader_uuid: "55897f8ef7f1482e9a381fff75600240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55897f8ef7f1482e9a381fff75600240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37249 } } }
I20260620 20:22:12.728564 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:12.731213 24211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:12.735545 24211 catalog_manager.cc:1347] Generated new cluster ID: db2ecf12968048e0b2b9463d968f2ef8
I20260620 20:22:12.735599 24211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:12.769356 24211 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:12.769629 24211 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:12.781913 24211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55897f8ef7f1482e9a381fff75600240: Generated new TSK 0
I20260620 20:22:12.782578 24211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:12.793732 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24124
2026-06-20T20:22:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:12 dist-test-slave-8z53 krb5kdc[24223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:12 dist-test-slave-8z53 krb5kdc[24223](info): set up 2 sockets
Jun 20 20:22:12 dist-test-slave-8z53 krb5kdc[24223](info): commencing operation
krb5kdc: starting...
W20260620 20:22:13.765642 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:13 dist-test-slave-8z53 krb5kdc[24223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:13.872008 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:13.970568 24236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:13.970780 24236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:13.970844 24236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:13.974360 24236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:13.974407 24236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:13.974426 24236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:13.974442 24236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:13.974483 24236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:13.979128 24236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:13.979947 24236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:13.980834 24236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:13.988250 24244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:13.988236 24241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:13.988226 24242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:13.988474 24236 server_base.cc:1034] running on GCE node
I20260620 20:22:13.989071 24236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:13.990073 24236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:13.991240 24236 hybrid_clock.cc:648] HybridClock initialized: now 1781986933991228 us; error 22 us; skew 500 ppm
Jun 20 20:22:13 dist-test-slave-8z53 krb5kdc[24223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:13.993621 24236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:13.994346 24236 webserver.cc:460] Webserver started at http://127.0.0.1:35477/ using document root <none> and password file <none>
I20260620 20:22:13.994807 24236 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:13.994884 24236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:13.995071 24236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:13.996896 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "03650b11c6f84dabbd18b1eaf259c705"
format_stamp: "Formatted at 2026-06-20 20:22:13 on dist-test-slave-8z53"
server_key: "32ccf5ec8fe53f82e89009def44e8d41"
server_key_iv: "e0ab3e9ab6e6616c8f6f41f1992cb823"
server_key_version: "encryptionkey@0"
I20260620 20:22:13.997475 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "03650b11c6f84dabbd18b1eaf259c705"
format_stamp: "Formatted at 2026-06-20 20:22:13 on dist-test-slave-8z53"
server_key: "32ccf5ec8fe53f82e89009def44e8d41"
server_key_iv: "e0ab3e9ab6e6616c8f6f41f1992cb823"
server_key_version: "encryptionkey@0"
I20260620 20:22:14.000523 24236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:14.002583 24251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:14.003607 24236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:22:14.003723 24236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "03650b11c6f84dabbd18b1eaf259c705"
format_stamp: "Formatted at 2026-06-20 20:22:13 on dist-test-slave-8z53"
server_key: "32ccf5ec8fe53f82e89009def44e8d41"
server_key_iv: "e0ab3e9ab6e6616c8f6f41f1992cb823"
server_key_version: "encryptionkey@0"
I20260620 20:22:14.003825 24236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:14.018141 24236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:14.020598 24236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:14.020781 24236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:14.026996 24236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33787
I20260620 20:22:14.027025 24303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33787 every 8 connection(s)
I20260620 20:22:14.027817 24236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:14.030176 24304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:14.034957 24304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705: Bootstrap starting.
I20260620 20:22:14.036574 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24236
I20260620 20:22:14.036711 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:14.037002 2991 external_mini_cluster.cc:1428] Setting key 18e6dfc6a5cf15a8c2ba23f4de64a76b
I20260620 20:22:14.037484 24304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:14.038345 24304 log.cc:824] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:14.040076 24304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705: No bootstrap required, opened a new log
I20260620 20:22:14.043169 24304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03650b11c6f84dabbd18b1eaf259c705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260620 20:22:14.043378 24304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:14.043434 24304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03650b11c6f84dabbd18b1eaf259c705, State: Initialized, Role: FOLLOWER
I20260620 20:22:14.043843 24304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03650b11c6f84dabbd18b1eaf259c705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260620 20:22:14.043975 24304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:14.044049 24304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:14.044147 24304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:14.044927 24304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03650b11c6f84dabbd18b1eaf259c705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260620 20:22:14.045324 24304 leader_election.cc:302] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03650b11c6f84dabbd18b1eaf259c705; no voters:
Jun 20 20:22:14 dist-test-slave-8z53 krb5kdc[24223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:22:14.045656 24304 leader_election.cc:288] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:14.045708 24309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:14.045941 24309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [term 1 LEADER]: Becoming Leader. State: Replica: 03650b11c6f84dabbd18b1eaf259c705, State: Running, Role: LEADER
I20260620 20:22:14.046276 24309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03650b11c6f84dabbd18b1eaf259c705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } }
I20260620 20:22:14.046830 24304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:14.047979 24310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03650b11c6f84dabbd18b1eaf259c705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03650b11c6f84dabbd18b1eaf259c705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } } }
I20260620 20:22:14.048019 24311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03650b11c6f84dabbd18b1eaf259c705. Latest consensus state: current_term: 1 leader_uuid: "03650b11c6f84dabbd18b1eaf259c705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03650b11c6f84dabbd18b1eaf259c705" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33787 } } }
I20260620 20:22:14.048107 24310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:14.048130 24311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705 [sys.catalog]: This master's current role is: LEADER
W20260620 20:22:14.051314 24324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:14.051399 24324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:14.051505 24318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:14.052134 24318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:14.052613 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:14.038127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:42021 (local address 127.0.0.1:33787)
0620 20:22:14.038765 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:14.038782 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:14.038825 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:22:14.039426 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:14.039442 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:14.039734 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:14.040032 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:14.040776 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:14.041850 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:14.042745 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:14.043206 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:14.043379 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:14.046305 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:14.046311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:14.046327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:14.046359 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:14.049316 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:14.049899 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:14.049906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:14.049911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:14.050026 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:14.050417 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:14.050421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:14.050423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:14.050895 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:14.051064 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:14.051563 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 20:22:14.051781 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":137,"threads_started":1}
I20260620 20:22:14.056301 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:14.056859 24318 catalog_manager.cc:1347] Generated new cluster ID: fa455b7124944430920a75d52dbaf455
I20260620 20:22:14.056957 24318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:14.065852 24318 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:14.066905 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:14.058493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:59113 (local address 127.0.0.1:33787)
0620 20:22:14.058588 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:14.058591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:14.058777 (+ 186us) server_negotiation.cc:407] Connection header received
0620 20:22:14.058873 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:14.058877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:14.058922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:14.058991 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:14.059610 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:14.060397 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:14.061486 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:14.061879 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:14.061969 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:14.063289 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:14.063292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:14.063296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:14.063320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:14.064944 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:14.065574 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:14.065577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:14.065578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:14.065657 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:14.066043 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:14.066046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:14.066048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:14.066259 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:14.066402 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:14.066668 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:22:14.066764 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 20:22:14.066910 24318 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:14.076617 24318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03650b11c6f84dabbd18b1eaf259c705: Generated new TSK 0
I20260620 20:22:14.077169 24318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:14.083386 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24236
W20260620 20:22:14.088263 24221 connection.cc:535] client connection to 127.0.0.1:33787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33787): Connection reset by peer (error 104)
2026-06-20T20:22:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:14 dist-test-slave-8z53 krb5kdc[24335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:14 dist-test-slave-8z53 krb5kdc[24335](info): set up 2 sockets
Jun 20 20:22:14 dist-test-slave-8z53 krb5kdc[24335](info): commencing operation
krb5kdc: starting...
W20260620 20:22:15.069635 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:15 dist-test-slave-8z53 krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:15.173663 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:15.271207 24348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:15.271427 24348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:15.271489 24348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:15.274993 24348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:15.275058 24348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:15.275092 24348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:15.275135 24348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:15.275174 24348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:15.279863 24348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:15.280633 24348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:15.281464 24348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:15.288869 24356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:15.288910 24354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:15.288906 24353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:15.289083 24348 server_base.cc:1034] running on GCE node
I20260620 20:22:15.289639 24348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:15.290534 24348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:15.291744 24348 hybrid_clock.cc:648] HybridClock initialized: now 1781986935291726 us; error 35 us; skew 500 ppm
Jun 20 20:22:15 dist-test-slave-8z53 krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:15.294153 24348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:15.294875 24348 webserver.cc:460] Webserver started at http://127.0.0.1:41049/ using document root <none> and password file <none>
I20260620 20:22:15.295327 24348 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:15.295380 24348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:15.295606 24348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:15.297420 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "631995b83bd446adae3404cb693c8064"
format_stamp: "Formatted at 2026-06-20 20:22:15 on dist-test-slave-8z53"
server_key: "7f1f203c8a9f46802b273aeaaf8fdbd7"
server_key_iv: "785a6f82af82d8359abde06d1b34ef61"
server_key_version: "encryptionkey@0"
I20260620 20:22:15.297904 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "631995b83bd446adae3404cb693c8064"
format_stamp: "Formatted at 2026-06-20 20:22:15 on dist-test-slave-8z53"
server_key: "7f1f203c8a9f46802b273aeaaf8fdbd7"
server_key_iv: "785a6f82af82d8359abde06d1b34ef61"
server_key_version: "encryptionkey@0"
I20260620 20:22:15.301003 24348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:15.303171 24363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:15.304229 24348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:22:15.304337 24348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "631995b83bd446adae3404cb693c8064"
format_stamp: "Formatted at 2026-06-20 20:22:15 on dist-test-slave-8z53"
server_key: "7f1f203c8a9f46802b273aeaaf8fdbd7"
server_key_iv: "785a6f82af82d8359abde06d1b34ef61"
server_key_version: "encryptionkey@0"
I20260620 20:22:15.304417 24348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:15.324788 24348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:15.327715 24348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:15.327876 24348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:15.334340 24348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44657
I20260620 20:22:15.334357 24415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44657 every 8 connection(s)
I20260620 20:22:15.335114 24348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:15.337647 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.338179 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24348
I20260620 20:22:15.338290 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:15.338553 2991 external_mini_cluster.cc:1428] Setting key 55350a16a0b56caa010d10c085a5f1fd
I20260620 20:22:15.343102 24416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064: Bootstrap starting.
I20260620 20:22:15.345088 24416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.345842 24416 log.cc:824] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:15.346565 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:15.339777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:52023 (local address 127.0.0.1:44657)
0620 20:22:15.340137 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:15.340150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:15.340180 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:15.340788 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:15.340802 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:15.341190 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:15.341535 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:15.341891 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.342730 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.343567 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.343979 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.344143 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:15.344886 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:15.344896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:15.344906 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:15.345254 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:15.345338 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:15.345359 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:15.345481 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:15.345651 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 20:22:15.345769 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260620 20:22:15.347822 24416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064: No bootstrap required, opened a new log
I20260620 20:22:15.349105 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:15.351447 24416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631995b83bd446adae3404cb693c8064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260620 20:22:15.351641 24416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.351701 24416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631995b83bd446adae3404cb693c8064, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.352057 24416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631995b83bd446adae3404cb693c8064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260620 20:22:15.352205 24416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:15.352278 24416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:15.352401 24416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.353188 24416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631995b83bd446adae3404cb693c8064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260620 20:22:15.353524 24416 leader_election.cc:302] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 631995b83bd446adae3404cb693c8064; no voters:
I20260620 20:22:15.353825 24416 leader_election.cc:288] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:15.353896 24426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:15.354362 24426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [term 1 LEADER]: Becoming Leader. State: Replica: 631995b83bd446adae3404cb693c8064, State: Running, Role: LEADER
I20260620 20:22:15.354709 24426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631995b83bd446adae3404cb693c8064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } }
I20260620 20:22:15.354830 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:15.350942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:59471 (local address 127.0.0.1:44657)
0620 20:22:15.351028 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:15.351031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:15.351217 (+ 186us) server_negotiation.cc:407] Connection header received
0620 20:22:15.351342 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:15.351345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:15.351385 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:15.351451 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:15.351683 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.352263 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.353055 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.353374 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.353479 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:15.354235 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:15.354238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:15.354241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:15.354253 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:15.354279 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:15.354296 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:15.354384 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:15.354684 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:22:15.354713 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 20:22:15.354938 24416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:15.356339 24428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 631995b83bd446adae3404cb693c8064. Latest consensus state: current_term: 1 leader_uuid: "631995b83bd446adae3404cb693c8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631995b83bd446adae3404cb693c8064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } } }
I20260620 20:22:15.356480 24428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:15.356339 24427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "631995b83bd446adae3404cb693c8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631995b83bd446adae3404cb693c8064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44657 } } }
I20260620 20:22:15.356865 24427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064 [sys.catalog]: This master's current role is: LEADER
W20260620 20:22:15.359129 24441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:15.359210 24441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:15.359287 24440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:15.359902 24440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:15.364645 24440 catalog_manager.cc:1347] Generated new cluster ID: 9ab7d79d3cdf41d8865035e4f7c1fde2
I20260620 20:22:15.364701 24440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:15.374850 24440 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:15.375900 24440 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:15.389461 24440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 631995b83bd446adae3404cb693c8064: Generated new TSK 0
I20260620 20:22:15.390087 24440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:15.399508 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24348
W20260620 20:22:15.404687 24333 connection.cc:535] client connection to 127.0.0.1:44657 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44657): Connection reset by peer (error 104)
2026-06-20T20:22:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:15 dist-test-slave-8z53 krb5kdc[24447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:15 dist-test-slave-8z53 krb5kdc[24447](info): set up 2 sockets
Jun 20 20:22:15 dist-test-slave-8z53 krb5kdc[24447](info): commencing operation
krb5kdc: starting...
W20260620 20:22:16.381311 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:16 dist-test-slave-8z53 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:22:16.486073 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:16.589310 24460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:16.589556 24460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:16.589594 24460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:16.593065 24460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:16.593107 24460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:16.593124 24460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:16.593138 24460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:16.593152 24460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:16.598131 24460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:16.598937 24460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:16.599869 24460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:16.607247 24465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:16.607288 24460 server_base.cc:1034] running on GCE node
W20260620 20:22:16.607306 24466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:16.607247 24468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:16.608094 24460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:16.609061 24460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:16.610212 24460 hybrid_clock.cc:648] HybridClock initialized: now 1781986936610188 us; error 34 us; skew 500 ppm
Jun 20 20:22:16 dist-test-slave-8z53 krb5kdc[24447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:16.612630 24460 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:22:16.613452 24460 webserver.cc:460] Webserver started at http://127.0.0.1:35863/ using document root <none> and password file <none>
I20260620 20:22:16.613927 24460 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:16.613981 24460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:16.614123 24460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:16.615922 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "09009fbf00e14ddda8ac866048b163eb"
format_stamp: "Formatted at 2026-06-20 20:22:16 on dist-test-slave-8z53"
server_key: "646c849f7245e21b23c7094f9e3d1427"
server_key_iv: "cd84b088909fa4530df75f529b274c66"
server_key_version: "encryptionkey@0"
I20260620 20:22:16.616429 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "09009fbf00e14ddda8ac866048b163eb"
format_stamp: "Formatted at 2026-06-20 20:22:16 on dist-test-slave-8z53"
server_key: "646c849f7245e21b23c7094f9e3d1427"
server_key_iv: "cd84b088909fa4530df75f529b274c66"
server_key_version: "encryptionkey@0"
I20260620 20:22:16.619863 24460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 20:22:16.622025 24475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:16.623129 24460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:22:16.623217 24460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "09009fbf00e14ddda8ac866048b163eb"
format_stamp: "Formatted at 2026-06-20 20:22:16 on dist-test-slave-8z53"
server_key: "646c849f7245e21b23c7094f9e3d1427"
server_key_iv: "cd84b088909fa4530df75f529b274c66"
server_key_version: "encryptionkey@0"
I20260620 20:22:16.623340 24460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:16.627381 24460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:16.629910 24460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:16.630080 24460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:16.636619 24460 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43979
I20260620 20:22:16.636644 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43979 every 8 connection(s)
I20260620 20:22:16.637461 24460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:16.639727 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:16.644307 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb: Bootstrap starting.
I20260620 20:22:16.644685 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24460
I20260620 20:22:16.644804 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:16.645049 2991 external_mini_cluster.cc:1428] Setting key 4e46aeb5586fc83109ed2365b4173e0d
I20260620 20:22:16.646620 24528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:16.647290 24528 log.cc:824] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:16.649232 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb: No bootstrap required, opened a new log
I20260620 20:22:16.649905 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:16.646127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:59429 (local address 127.0.0.1:43979)
0620 20:22:16.646555 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:16.646570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:16.646605 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:22:16.647275 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:16.647291 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:16.647642 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:16.648014 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:16.648727 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:16.648731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:16.648760 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:16.648770 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:16.648820 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:16.648838 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:16.648922 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:16.649044 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:22:16.649197 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":103,"threads_started":1}
I20260620 20:22:16.652364 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:16.652347 24528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09009fbf00e14ddda8ac866048b163eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260620 20:22:16.652565 24528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:16.652611 24528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09009fbf00e14ddda8ac866048b163eb, State: Initialized, Role: FOLLOWER
I20260620 20:22:16.653007 24528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09009fbf00e14ddda8ac866048b163eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260620 20:22:16.653139 24528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:16.653200 24528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:16.653302 24528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:16.654316 24528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09009fbf00e14ddda8ac866048b163eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260620 20:22:16.654665 24528 leader_election.cc:302] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09009fbf00e14ddda8ac866048b163eb; no voters:
I20260620 20:22:16.654996 24528 leader_election.cc:288] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:16.655045 24538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:16.655243 24538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [term 1 LEADER]: Becoming Leader. State: Replica: 09009fbf00e14ddda8ac866048b163eb, State: Running, Role: LEADER
I20260620 20:22:16.655481 24538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09009fbf00e14ddda8ac866048b163eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260620 20:22:16.656045 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:16.654488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:37669 (local address 127.0.0.1:43979)
0620 20:22:16.654612 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:16.654616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:16.654861 (+ 245us) server_negotiation.cc:407] Connection header received
0620 20:22:16.654945 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:16.654948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:16.654980 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:16.655052 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:16.655685 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:16.655687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:16.655689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:16.655698 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:16.655721 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:16.655735 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:16.655789 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:16.655930 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:22:16.655959 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:22:16.656072 24528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:16.657078 24539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09009fbf00e14ddda8ac866048b163eb. Latest consensus state: current_term: 1 leader_uuid: "09009fbf00e14ddda8ac866048b163eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09009fbf00e14ddda8ac866048b163eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } } }
I20260620 20:22:16.657195 24539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:16.657310 24540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09009fbf00e14ddda8ac866048b163eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09009fbf00e14ddda8ac866048b163eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } } }
I20260620 20:22:16.657402 24540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:16.657622 24543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:16.660362 24543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:16.665448 24543 catalog_manager.cc:1347] Generated new cluster ID: da4832b382b94aa3b08bd1756ff3fb74
I20260620 20:22:16.665512 24543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:16.684109 24543 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:16.684386 24543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:16.690559 24543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09009fbf00e14ddda8ac866048b163eb: Generated new TSK 0
I20260620 20:22:16.691138 24543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:16.696846 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24460
2026-06-20T20:22:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:16 dist-test-slave-8z53 krb5kdc[24559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:16 dist-test-slave-8z53 krb5kdc[24559](info): set up 2 sockets
Jun 20 20:22:16 dist-test-slave-8z53 krb5kdc[24559](info): commencing operation
krb5kdc: starting...
W20260620 20:22:17.681607 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:17 dist-test-slave-8z53 krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:17.786458 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:35499
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:35499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:17.887358 24572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:17.887610 24572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:17.887679 24572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:17.891390 24572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:17.891431 24572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:17.891448 24572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:17.891462 24572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:17.891475 24572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:17.896005 24572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:35499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:35499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:17.896834 24572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:17.897742 24572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:17.904983 24578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:17.905019 24572 server_base.cc:1034] running on GCE node
W20260620 20:22:17.905078 24577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:17.905073 24580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:17.905706 24572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:17.906620 24572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:17.907785 24572 hybrid_clock.cc:648] HybridClock initialized: now 1781986937907770 us; error 33 us; skew 500 ppm
Jun 20 20:22:17 dist-test-slave-8z53 krb5kdc[24559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:17.913964 24572 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:17.914652 24572 webserver.cc:460] Webserver started at http://127.2.235.254:45961/ using document root <none> and password file <none>
I20260620 20:22:17.915148 24572 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:17.915233 24572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:17.915436 24572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:17.917222 24572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "dfc387ef094d4618bb230af07fd11ce1"
format_stamp: "Formatted at 2026-06-20 20:22:17 on dist-test-slave-8z53"
server_key: "91dfe1a0a74175e4da60626aff9b1880"
server_key_iv: "5b5fe20ea51c106be738fafe271591ed"
server_key_version: "encryptionkey@0"
I20260620 20:22:17.917838 24572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "dfc387ef094d4618bb230af07fd11ce1"
format_stamp: "Formatted at 2026-06-20 20:22:17 on dist-test-slave-8z53"
server_key: "91dfe1a0a74175e4da60626aff9b1880"
server_key_iv: "5b5fe20ea51c106be738fafe271591ed"
server_key_version: "encryptionkey@0"
I20260620 20:22:17.920828 24572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:17.922885 24587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:17.923909 24572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 20:22:17.924052 24572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "dfc387ef094d4618bb230af07fd11ce1"
format_stamp: "Formatted at 2026-06-20 20:22:17 on dist-test-slave-8z53"
server_key: "91dfe1a0a74175e4da60626aff9b1880"
server_key_iv: "5b5fe20ea51c106be738fafe271591ed"
server_key_version: "encryptionkey@0"
I20260620 20:22:17.924155 24572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:17.939234 24572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:17.942179 24572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:17.942361 24572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:17.948843 24572 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:35499
I20260620 20:22:17.948856 24639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:35499 every 8 connection(s)
I20260620 20:22:17.949756 24572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:17.951056 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24572
I20260620 20:22:17.951200 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:17.951448 2991 external_mini_cluster.cc:1428] Setting key bbf5cb8a8d6b5fcef04a4840d5b132aa
I20260620 20:22:17.952409 24640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:17.957479 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1: Bootstrap starting.
Jun 20 20:22:17 dist-test-slave-8z53 krb5kdc[24559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:17.959545 24640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:17.960305 24640 log.cc:824] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:17.962227 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1: No bootstrap required, opened a new log
I20260620 20:22:17.964399 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:17.952673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48665 (local address 127.2.235.254:35499)
0620 20:22:17.953069 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:17.953082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:17.953113 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:22:17.953738 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:17.953754 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:17.954046 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:17.954368 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:17.955133 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.956102 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.956958 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.957482 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.957649 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:17.960177 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:17.960182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:17.960193 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:17.960215 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:17.961845 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.962383 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.962387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.962391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.962499 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.962855 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.962857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.962858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.963160 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:17.963303 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:17.963608 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 20:22:17.963814 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260620 20:22:17.965725 24640 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc387ef094d4618bb230af07fd11ce1" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35499 } }
I20260620 20:22:17.965902 24640 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:17.965955 24640 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc387ef094d4618bb230af07fd11ce1, State: Initialized, Role: FOLLOWER
I20260620 20:22:17.966403 24640 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc387ef094d4618bb230af07fd11ce1" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35499 } }
I20260620 20:22:17.966565 24640 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:17.966634 24640 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:17.966733 24640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:17.967131 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:17.967729 24640 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc387ef094d4618bb230af07fd11ce1" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35499 } }
I20260620 20:22:17.968070 24640 leader_election.cc:302] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc387ef094d4618bb230af07fd11ce1; no voters:
I20260620 20:22:17.968379 24640 leader_election.cc:288] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:17.968488 24645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:17.968736 24645 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [term 1 LEADER]: Becoming Leader. State: Replica: dfc387ef094d4618bb230af07fd11ce1, State: Running, Role: LEADER
I20260620 20:22:17.969035 24645 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc387ef094d4618bb230af07fd11ce1" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35499 } }
I20260620 20:22:17.969717 24640 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:17.970166 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc387ef094d4618bb230af07fd11ce1. Latest consensus state: current_term: 1 leader_uuid: "dfc387ef094d4618bb230af07fd11ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc387ef094d4618bb230af07fd11ce1" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35499 } } }
I20260620 20:22:17.970289 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:17.970575 24650 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc387ef094d4618bb230af07fd11ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc387ef094d4618bb230af07fd11ce1" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 35499 } } }
I20260620 20:22:17.970741 24650 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:17.971902 24655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:17.974224 24655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:17.979389 24655 catalog_manager.cc:1347] Generated new cluster ID: 74ef0798fcb84cb984fbed29bc1c12cb
I20260620 20:22:17.979501 24655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:17.983072 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:17.969634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56745 (local address 127.2.235.254:35499)
0620 20:22:17.969739 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:17.969742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:17.975734 (+ 5992us) server_negotiation.cc:407] Connection header received
0620 20:22:17.975872 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:17.975875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:17.975914 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:17.975992 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:17.976594 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.977080 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.978147 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.978519 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.978625 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:17.980007 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:17.980011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:17.980013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:17.980040 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:17.981464 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.981897 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.981902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.981908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.981972 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.982275 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.982278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.982280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.982426 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:17.982566 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:17.982831 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:22:17.982914 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:22:17.988317 24655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:17.989144 24655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:17.997526 24655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc387ef094d4618bb230af07fd11ce1: Generated new TSK 0
I20260620 20:22:17.998090 24655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:18.009307 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24572
W20260620 20:22:18.013912 24557 connection.cc:535] client connection to 127.2.235.254:35499 recv error: Network error: failed to read from TLS socket (remote: 127.2.235.254:35499): Connection reset by peer (error 104)
2026-06-20T20:22:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:18 dist-test-slave-8z53 krb5kdc[24671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:18 dist-test-slave-8z53 krb5kdc[24671](info): set up 2 sockets
Jun 20 20:22:18 dist-test-slave-8z53 krb5kdc[24671](info): commencing operation
krb5kdc: starting...
W20260620 20:22:18.973294 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:19 dist-test-slave-8z53 krb5kdc[24671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:19.076731 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41449
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:35435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:19.178123 24684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:19.178320 24684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:19.178356 24684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:19.182117 24684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:19.182159 24684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:19.182176 24684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:19.182189 24684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:19.182253 24684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:19.186862 24684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:35435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:19.187675 24684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:19.188544 24684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:19.196213 24689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:19.196202 24690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:19.196352 24684 server_base.cc:1034] running on GCE node
W20260620 20:22:19.196516 24692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:19.197019 24684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:19.197996 24684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:19.199152 24684 hybrid_clock.cc:648] HybridClock initialized: now 1781986939199130 us; error 34 us; skew 500 ppm
Jun 20 20:22:19 dist-test-slave-8z53 krb5kdc[24671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986939, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:19.201638 24684 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:19.202447 24684 webserver.cc:460] Webserver started at http://127.2.235.254:41709/ using document root <none> and password file <none>
I20260620 20:22:19.202970 24684 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:19.203058 24684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:19.203270 24684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:19.205109 24684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "01fd2755a2c84e02a829527bf6522397"
format_stamp: "Formatted at 2026-06-20 20:22:19 on dist-test-slave-8z53"
server_key: "b1164738b466a5ea50fb541baa5766b9"
server_key_iv: "65f733cbf2fd6d9e938bdefba312e288"
server_key_version: "encryptionkey@0"
I20260620 20:22:19.205689 24684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "01fd2755a2c84e02a829527bf6522397"
format_stamp: "Formatted at 2026-06-20 20:22:19 on dist-test-slave-8z53"
server_key: "b1164738b466a5ea50fb541baa5766b9"
server_key_iv: "65f733cbf2fd6d9e938bdefba312e288"
server_key_version: "encryptionkey@0"
I20260620 20:22:19.209157 24684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:19.211325 24699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:19.212327 24684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:19.212450 24684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "01fd2755a2c84e02a829527bf6522397"
format_stamp: "Formatted at 2026-06-20 20:22:19 on dist-test-slave-8z53"
server_key: "b1164738b466a5ea50fb541baa5766b9"
server_key_iv: "65f733cbf2fd6d9e938bdefba312e288"
server_key_version: "encryptionkey@0"
I20260620 20:22:19.212555 24684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:19.228425 24684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:19.231266 24684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:19.231468 24684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:19.237993 24684 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:41449
I20260620 20:22:19.238014 24751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:41449 every 8 connection(s)
I20260620 20:22:19.238806 24684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:19.241631 24752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:19.242745 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24684
I20260620 20:22:19.242893 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:19.243148 2991 external_mini_cluster.cc:1428] Setting key 9b3c6d129e4c8fc07ad17e31807d4c93
I20260620 20:22:19.246879 24752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397: Bootstrap starting.
I20260620 20:22:19.248775 24752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:19.249495 24752 log.cc:824] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:19.251284 24752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397: No bootstrap required, opened a new log
I20260620 20:22:19.251317 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:19.244352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54649 (local address 127.2.235.254:41449)
0620 20:22:19.244748 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:19.244760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:19.244794 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:22:19.245420 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:19.245436 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:19.245718 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:19.246013 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:19.246382 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:19.247367 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:19.248221 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:19.248635 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:19.248785 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:19.249652 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:19.249662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:19.249676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:19.250170 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:19.250229 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:19.250269 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:19.250400 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:19.250574 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 20:22:19.250708 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260620 20:22:19.253837 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:19.254861 24752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fd2755a2c84e02a829527bf6522397" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41449 } }
I20260620 20:22:19.255143 24752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:19.255219 24752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fd2755a2c84e02a829527bf6522397, State: Initialized, Role: FOLLOWER
I20260620 20:22:19.255677 24752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fd2755a2c84e02a829527bf6522397" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41449 } }
I20260620 20:22:19.255841 24752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:19.255964 24752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:19.256098 24752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:19.256870 24752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fd2755a2c84e02a829527bf6522397" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41449 } }
I20260620 20:22:19.257184 24752 leader_election.cc:302] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01fd2755a2c84e02a829527bf6522397; no voters:
I20260620 20:22:19.257507 24752 leader_election.cc:288] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:19.257584 24762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:19.257747 24762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [term 1 LEADER]: Becoming Leader. State: Replica: 01fd2755a2c84e02a829527bf6522397, State: Running, Role: LEADER
I20260620 20:22:19.258073 24762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fd2755a2c84e02a829527bf6522397" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41449 } }
I20260620 20:22:19.258632 24752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:19.259681 24764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01fd2755a2c84e02a829527bf6522397. Latest consensus state: current_term: 1 leader_uuid: "01fd2755a2c84e02a829527bf6522397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fd2755a2c84e02a829527bf6522397" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41449 } } }
I20260620 20:22:19.260002 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:19.255812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49349 (local address 127.2.235.254:41449)
0620 20:22:19.255933 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:19.255937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:19.256078 (+ 141us) server_negotiation.cc:407] Connection header received
0620 20:22:19.256182 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:19.256185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:19.256220 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:19.256313 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:19.256558 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:19.257126 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:19.258208 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:19.258474 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:19.258526 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:19.259469 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:19.259475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:19.259479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:19.259494 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:19.259522 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:19.259537 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:19.259603 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:19.259858 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:22:19.259889 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:22:19.260196 24764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:19.260082 24763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01fd2755a2c84e02a829527bf6522397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01fd2755a2c84e02a829527bf6522397" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41449 } } }
I20260620 20:22:19.260316 24763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:19.261076 24773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:19.263823 24773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:19.268996 24773 catalog_manager.cc:1347] Generated new cluster ID: 6115ec0400924d90bee1eea9752c7b8a
I20260620 20:22:19.269052 24773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:19.280159 24773 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:19.281157 24773 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:19.290441 24773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01fd2755a2c84e02a829527bf6522397: Generated new TSK 0
I20260620 20:22:19.291039 24773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:19.302809 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24684
W20260620 20:22:19.307889 24669 connection.cc:535] client connection to 127.2.235.254:41449 recv error: Network error: failed to read from TLS socket (remote: 127.2.235.254:41449): Connection reset by peer (error 104)
2026-06-20T20:22:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:19 dist-test-slave-8z53 krb5kdc[24783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:19 dist-test-slave-8z53 krb5kdc[24783](info): set up 2 sockets
Jun 20 20:22:19 dist-test-slave-8z53 krb5kdc[24783](info): commencing operation
krb5kdc: starting...
W20260620 20:22:20.285638 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:20 dist-test-slave-8z53 krb5kdc[24783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:20.392158 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:45009
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:20.494199 24796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:20.494463 24796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:20.494560 24796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:20.498395 24796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:20.498461 24796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:20.498493 24796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:20.498545 24796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:20.498579 24796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:20.503132 24796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:45009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:20.503938 24796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:20.504750 24796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:22:20.512341 24796 server_base.cc:1034] running on GCE node
W20260620 20:22:20.512300 24802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:20.512313 24801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:20.512285 24804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:20.512924 24796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:20.513814 24796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:20.514982 24796 hybrid_clock.cc:648] HybridClock initialized: now 1781986940514966 us; error 25 us; skew 500 ppm
Jun 20 20:22:20 dist-test-slave-8z53 krb5kdc[24783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986940, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:20.517525 24796 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:20.518326 24796 webserver.cc:460] Webserver started at http://127.2.235.254:38191/ using document root <none> and password file <none>
I20260620 20:22:20.518818 24796 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:20.518900 24796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:20.519109 24796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:20.520787 24796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf"
format_stamp: "Formatted at 2026-06-20 20:22:20 on dist-test-slave-8z53"
server_key: "0c9a7406b8d436887e4b15d22a6a8b6b"
server_key_iv: "303bc374f9865413b5c614a194fc6444"
server_key_version: "encryptionkey@0"
I20260620 20:22:20.521348 24796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf"
format_stamp: "Formatted at 2026-06-20 20:22:20 on dist-test-slave-8z53"
server_key: "0c9a7406b8d436887e4b15d22a6a8b6b"
server_key_iv: "303bc374f9865413b5c614a194fc6444"
server_key_version: "encryptionkey@0"
I20260620 20:22:20.524437 24796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:22:20.526503 24811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:20.527539 24796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:22:20.527667 24796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf"
format_stamp: "Formatted at 2026-06-20 20:22:20 on dist-test-slave-8z53"
server_key: "0c9a7406b8d436887e4b15d22a6a8b6b"
server_key_iv: "303bc374f9865413b5c614a194fc6444"
server_key_version: "encryptionkey@0"
I20260620 20:22:20.527766 24796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:20.531808 24796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:20.534548 24796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:20.534713 24796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:20.541149 24796 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:45009
I20260620 20:22:20.541172 24863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:45009 every 8 connection(s)
I20260620 20:22:20.541944 24796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:20.544236 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:20.546780 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24796
I20260620 20:22:20.546897 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:20.547179 2991 external_mini_cluster.cc:1428] Setting key 26b05e2c92fe1ca254613ff80040a141
I20260620 20:22:20.549683 24864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf: Bootstrap starting.
I20260620 20:22:20.551582 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:20.548384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49191 (local address 127.2.235.254:45009)
0620 20:22:20.548799 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:20.548812 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:20.548842 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:20.549440 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:20.549456 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:20.549685 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:20.550005 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:20.550637 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:20.550642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:20.550654 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:20.550664 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:20.550732 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:20.550748 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:20.550870 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:20.550931 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 20:22:20.551054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260620 20:22:20.551831 24864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:20.552553 24864 log.cc:824] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:20.554165 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:20.554508 24864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf: No bootstrap required, opened a new log
I20260620 20:22:20.557636 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:20.556045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43689 (local address 127.2.235.254:45009)
0620 20:22:20.556126 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:20.556129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:20.556336 (+ 207us) server_negotiation.cc:407] Connection header received
0620 20:22:20.556440 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:20.556443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:20.556470 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:20.556548 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:20.557219 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:20.557222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:20.557225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:20.557235 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:20.557308 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:20.557321 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:20.557421 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:20.557495 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:22:20.557530 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 20:22:20.558470 24864 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 45009 } }
I20260620 20:22:20.558661 24864 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:20.558748 24864 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca80cfb8fa74a8aac6b9df1fbc1c4bf, State: Initialized, Role: FOLLOWER
I20260620 20:22:20.559125 24864 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 45009 } }
I20260620 20:22:20.559296 24864 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:20.559412 24864 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:20.559527 24864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:20.560364 24864 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 45009 } }
I20260620 20:22:20.560645 24864 leader_election.cc:302] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca80cfb8fa74a8aac6b9df1fbc1c4bf; no voters:
I20260620 20:22:20.560953 24864 leader_election.cc:288] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:20.561067 24874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:20.561364 24874 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [term 1 LEADER]: Becoming Leader. State: Replica: dca80cfb8fa74a8aac6b9df1fbc1c4bf, State: Running, Role: LEADER
I20260620 20:22:20.561659 24874 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 45009 } }
I20260620 20:22:20.562213 24864 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:20.563194 24875 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca80cfb8fa74a8aac6b9df1fbc1c4bf. Latest consensus state: current_term: 1 leader_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 45009 } } }
I20260620 20:22:20.563311 24875 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:20.563676 24883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:20.563642 24876 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca80cfb8fa74a8aac6b9df1fbc1c4bf" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 45009 } } }
I20260620 20:22:20.564242 24876 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:20.566136 24883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:20.571681 24883 catalog_manager.cc:1347] Generated new cluster ID: 4d05bc15a0f64a709a7811d102384749
I20260620 20:22:20.571790 24883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:20.577409 24883 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:20.577694 24883 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:20.581298 24883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca80cfb8fa74a8aac6b9df1fbc1c4bf: Generated new TSK 0
I20260620 20:22:20.581887 24883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:20.593945 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24796
2026-06-20T20:22:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:20 dist-test-slave-8z53 krb5kdc[24895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:20 dist-test-slave-8z53 krb5kdc[24895](info): set up 2 sockets
Jun 20 20:22:20 dist-test-slave-8z53 krb5kdc[24895](info): commencing operation
krb5kdc: starting...
W20260620 20:22:21.573694 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:21 dist-test-slave-8z53 krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:21.681816 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44429
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:21.780781 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:21.781018 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:21.781097 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:21.784859 24908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:21.784933 24908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:21.784968 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:21.785000 24908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:21.785028 24908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:21.789700 24908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:21.790495 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:21.791312 24908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:22:21.798657 24908 server_base.cc:1034] running on GCE node
W20260620 20:22:21.798588 24914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:21.798581 24913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:21.798597 24916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:21.799322 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:21.800150 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:21.801323 24908 hybrid_clock.cc:648] HybridClock initialized: now 1781986941801306 us; error 28 us; skew 500 ppm
Jun 20 20:22:21 dist-test-slave-8z53 krb5kdc[24895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986941, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:21.803675 24908 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:21.804503 24908 webserver.cc:460] Webserver started at http://127.2.235.254:35167/ using document root <none> and password file <none>
I20260620 20:22:21.804965 24908 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:21.805047 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:21.805243 24908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:21.807031 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "a4ff8be3213a4bcf8098a193da0738d3"
format_stamp: "Formatted at 2026-06-20 20:22:21 on dist-test-slave-8z53"
server_key: "8240e8afc645047fb9b30011e59e7bb3"
server_key_iv: "e5594e74c0f6206e7d9b8f7d2a2cee74"
server_key_version: "encryptionkey@0"
I20260620 20:22:21.807554 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "a4ff8be3213a4bcf8098a193da0738d3"
format_stamp: "Formatted at 2026-06-20 20:22:21 on dist-test-slave-8z53"
server_key: "8240e8afc645047fb9b30011e59e7bb3"
server_key_iv: "e5594e74c0f6206e7d9b8f7d2a2cee74"
server_key_version: "encryptionkey@0"
I20260620 20:22:21.810729 24908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:22:21.812862 24923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:21.814025 24908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:21.814174 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "a4ff8be3213a4bcf8098a193da0738d3"
format_stamp: "Formatted at 2026-06-20 20:22:21 on dist-test-slave-8z53"
server_key: "8240e8afc645047fb9b30011e59e7bb3"
server_key_iv: "e5594e74c0f6206e7d9b8f7d2a2cee74"
server_key_version: "encryptionkey@0"
I20260620 20:22:21.814287 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:21.824008 24908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:21.826639 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:21.826833 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:21.833586 24908 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:44429
I20260620 20:22:21.833626 24975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:44429 every 8 connection(s)
I20260620 20:22:21.834421 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:21.836154 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 24908
I20260620 20:22:21.836284 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:21.836570 2991 external_mini_cluster.cc:1428] Setting key a86ac285ec6f2e5593992a3bcfb45199
I20260620 20:22:21.837337 24976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:21.843570 24976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3: Bootstrap starting.
Jun 20 20:22:21 dist-test-slave-8z53 krb5kdc[24895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:21.845907 24976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:21.846571 24976 log.cc:824] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:21.848591 24976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3: No bootstrap required, opened a new log
I20260620 20:22:21.850627 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:21.837810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52593 (local address 127.2.235.254:44429)
0620 20:22:21.838229 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:21.838251 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:21.838281 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:21.838828 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:21.838843 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:21.839139 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:21.839407 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:21.840269 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:21.841391 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:21.842513 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:21.843004 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:21.843115 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:21.845749 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:21.845755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:21.845767 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:21.845792 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:21.847767 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:21.848242 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:21.848247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:21.848289 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:21.848388 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:21.848791 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:21.848794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:21.848796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:21.849199 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:21.849448 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:21.849699 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 20:22:21.850022 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260620 20:22:21.852111 24976 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ff8be3213a4bcf8098a193da0738d3" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44429 } }
I20260620 20:22:21.852324 24976 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:21.852384 24976 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ff8be3213a4bcf8098a193da0738d3, State: Initialized, Role: FOLLOWER
I20260620 20:22:21.852799 24976 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ff8be3213a4bcf8098a193da0738d3" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44429 } }
I20260620 20:22:21.852907 24976 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:21.852980 24976 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:21.853068 24976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:21.853159 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:21.854020 24976 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ff8be3213a4bcf8098a193da0738d3" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44429 } }
I20260620 20:22:21.854336 24976 leader_election.cc:302] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ff8be3213a4bcf8098a193da0738d3; no voters:
I20260620 20:22:21.854673 24976 leader_election.cc:288] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:21.854759 24985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:21.854995 24985 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [term 1 LEADER]: Becoming Leader. State: Replica: a4ff8be3213a4bcf8098a193da0738d3, State: Running, Role: LEADER
I20260620 20:22:21.855355 24985 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ff8be3213a4bcf8098a193da0738d3" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44429 } }
I20260620 20:22:21.855911 24976 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:21.856433 24987 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ff8be3213a4bcf8098a193da0738d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ff8be3213a4bcf8098a193da0738d3" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44429 } } }
I20260620 20:22:21.856433 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ff8be3213a4bcf8098a193da0738d3. Latest consensus state: current_term: 1 leader_uuid: "a4ff8be3213a4bcf8098a193da0738d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ff8be3213a4bcf8098a193da0738d3" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44429 } } }
I20260620 20:22:21.856576 24987 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:21.856576 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:21.857549 24992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:21.860457 24992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:21.863494 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:21.855363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56775 (local address 127.2.235.254:44429)
0620 20:22:21.855457 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:21.855461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:21.855714 (+ 253us) server_negotiation.cc:407] Connection header received
0620 20:22:21.855837 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:21.855840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:21.855881 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:21.855952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:21.856870 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:21.857721 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:21.858775 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:21.859132 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:21.859223 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:21.860692 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:21.860696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:21.860699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:21.860721 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:21.862027 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:21.862422 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:21.862425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:21.862427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:21.862498 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:21.862755 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:21.862759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:21.862761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:21.862917 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:21.863036 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:21.863243 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 20:22:21.863352 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:22:21.865682 24992 catalog_manager.cc:1347] Generated new cluster ID: ba7b08ecacd84bea89f5b0e3f1ca6383
I20260620 20:22:21.865739 24992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:21.893797 24992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:21.894600 24992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:21.898397 24992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ff8be3213a4bcf8098a193da0738d3: Generated new TSK 0
I20260620 20:22:21.899116 24992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:21.904742 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 24908
2026-06-20T20:22:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:21 dist-test-slave-8z53 krb5kdc[25007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:21 dist-test-slave-8z53 krb5kdc[25007](info): set up 2 sockets
Jun 20 20:22:21 dist-test-slave-8z53 krb5kdc[25007](info): commencing operation
krb5kdc: starting...
W20260620 20:22:22.877549 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:22 dist-test-slave-8z53 krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:22.985868 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43797
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:23.084138 25020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:23.084380 25020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:23.084470 25020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:23.088305 25020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:23.088382 25020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:23.088418 25020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:23.088469 25020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:23.088505 25020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:23.093060 25020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:43797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:23.093904 25020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:23.094766 25020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:23.102644 25028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:23.102669 25025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:23.102653 25026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:23.102926 25020 server_base.cc:1034] running on GCE node
I20260620 20:22:23.103454 25020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:23.104300 25020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:23.105451 25020 hybrid_clock.cc:648] HybridClock initialized: now 1781986943105436 us; error 26 us; skew 500 ppm
Jun 20 20:22:23 dist-test-slave-8z53 krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986943, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:23.112632 25020 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:23.113466 25020 webserver.cc:460] Webserver started at http://127.2.235.254:42289/ using document root <none> and password file <none>
I20260620 20:22:23.113936 25020 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:23.113991 25020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:23.114215 25020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:23.116084 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "8f6b8400f2b0471c9d2a0e9405e20621"
format_stamp: "Formatted at 2026-06-20 20:22:23 on dist-test-slave-8z53"
server_key: "3e8f429656225713b8047f43d743b8c9"
server_key_iv: "c63d3af95a3849d774342148344daf5b"
server_key_version: "encryptionkey@0"
I20260620 20:22:23.116632 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "8f6b8400f2b0471c9d2a0e9405e20621"
format_stamp: "Formatted at 2026-06-20 20:22:23 on dist-test-slave-8z53"
server_key: "3e8f429656225713b8047f43d743b8c9"
server_key_iv: "c63d3af95a3849d774342148344daf5b"
server_key_version: "encryptionkey@0"
I20260620 20:22:23.119947 25020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:22:23.122004 25035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:23.123044 25020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:23.123183 25020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "8f6b8400f2b0471c9d2a0e9405e20621"
format_stamp: "Formatted at 2026-06-20 20:22:23 on dist-test-slave-8z53"
server_key: "3e8f429656225713b8047f43d743b8c9"
server_key_iv: "c63d3af95a3849d774342148344daf5b"
server_key_version: "encryptionkey@0"
I20260620 20:22:23.123293 25020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:23.142694 25020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:23.145633 25020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:23.145816 25020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:23.152328 25020 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:43797
I20260620 20:22:23.152345 25087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:43797 every 8 connection(s)
I20260620 20:22:23.153234 25020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:23.155637 25088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:23.160583 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25020
I20260620 20:22:23.160682 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:23.160676 25088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621: Bootstrap starting.
I20260620 20:22:23.160982 2991 external_mini_cluster.cc:1428] Setting key 14a568bc7c087d39922e5569fd6992e3
I20260620 20:22:23.163141 25088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:23.163959 25088 log.cc:824] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:23.165874 25088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621: No bootstrap required, opened a new log
I20260620 20:22:23.169961 25088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43797 } }
I20260620 20:22:23.170190 25088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:23.170255 25088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6b8400f2b0471c9d2a0e9405e20621, State: Initialized, Role: FOLLOWER
I20260620 20:22:23.170285 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:23.162269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44913 (local address 127.2.235.254:43797)
0620 20:22:23.162665 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:23.162682 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:23.162720 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:22:23.163542 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:23.163563 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:23.163950 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:23.164272 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:23.164559 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:23.165868 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:23.166739 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:23.167427 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:23.167591 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:23.168319 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:23.168326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:23.168345 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:23.168867 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:23.168927 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:23.168954 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:23.169046 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:23.169215 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:22:23.169444 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260620 20:22:23.170634 25088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43797 } }
I20260620 20:22:23.170784 25088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:23.170864 25088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:23.170960 25088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:23.171849 25088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43797 } }
I20260620 20:22:23.172164 25088 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6b8400f2b0471c9d2a0e9405e20621; no voters:
I20260620 20:22:23.172449 25088 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:23.172538 25093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:23.172730 25093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6b8400f2b0471c9d2a0e9405e20621, State: Running, Role: LEADER
I20260620 20:22:23.172979 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:23.173053 25093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43797 } }
I20260620 20:22:23.173568 25088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:23.174471 25095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6b8400f2b0471c9d2a0e9405e20621. Latest consensus state: current_term: 1 leader_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43797 } } }
I20260620 20:22:23.174600 25095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:23.174902 25094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b8400f2b0471c9d2a0e9405e20621" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 43797 } } }
I20260620 20:22:23.174978 25094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:23.175848 25106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:23.178797 25106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:23.179209 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:23.175156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38517 (local address 127.2.235.254:43797)
0620 20:22:23.175258 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:23.175263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:23.175849 (+ 586us) server_negotiation.cc:407] Connection header received
0620 20:22:23.176018 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:23.176021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:23.176068 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:23.176135 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:23.176382 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:23.176851 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:23.177726 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:23.177950 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:23.178058 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:23.178768 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:23.178772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:23.178774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:23.178785 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:23.178808 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:23.178827 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:23.178907 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:23.179044 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:22:23.179087 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:22:23.184410 25106 catalog_manager.cc:1347] Generated new cluster ID: 3a043797124943daa11faab52b5c3ce7
I20260620 20:22:23.184505 25106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:23.195005 25106 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:23.196126 25106 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:23.231220 25106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6b8400f2b0471c9d2a0e9405e20621: Generated new TSK 0
I20260620 20:22:23.231829 25106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:23.238102 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25020
2026-06-20T20:22:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:23 dist-test-slave-8z53 krb5kdc[25119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:23 dist-test-slave-8z53 krb5kdc[25119](info): set up 2 sockets
Jun 20 20:22:23 dist-test-slave-8z53 krb5kdc[25119](info): commencing operation
krb5kdc: starting...
W20260620 20:22:24.233441 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:24 dist-test-slave-8z53 krb5kdc[25119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:24.339586 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33377
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:33377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:24.438169 25132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:24.438416 25132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:24.438480 25132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:24.442281 25132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:24.442353 25132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:24.442389 25132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:24.442421 25132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:24.442454 25132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:24.447100 25132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:33377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:24.448012 25132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:24.448871 25132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:24.456127 25138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:24.456132 25137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:24.456187 25132 server_base.cc:1034] running on GCE node
W20260620 20:22:24.456132 25140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:24.456787 25132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:24.457836 25132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:24.459064 25132 hybrid_clock.cc:648] HybridClock initialized: now 1781986944459020 us; error 39 us; skew 500 ppm
Jun 20 20:22:24 dist-test-slave-8z53 krb5kdc[25119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:24.461521 25132 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:24.462306 25132 webserver.cc:460] Webserver started at http://127.2.235.254:35213/ using document root <none> and password file <none>
I20260620 20:22:24.462832 25132 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:24.462913 25132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:24.463173 25132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:24.465018 25132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "1bc8211ba30740b9800b76d79e5c9641"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-8z53"
server_key: "b5f9bccb6c44d8a55000efc4ce33623d"
server_key_iv: "00372e0943f7ca20073115a7779d0f85"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.465564 25132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "1bc8211ba30740b9800b76d79e5c9641"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-8z53"
server_key: "b5f9bccb6c44d8a55000efc4ce33623d"
server_key_iv: "00372e0943f7ca20073115a7779d0f85"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.468875 25132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:22:24.471053 25147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.472110 25132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:24.472235 25132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "1bc8211ba30740b9800b76d79e5c9641"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-8z53"
server_key: "b5f9bccb6c44d8a55000efc4ce33623d"
server_key_iv: "00372e0943f7ca20073115a7779d0f85"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.472330 25132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:24.476401 25132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:24.479036 25132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:24.479210 25132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:24.485385 25132 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:33377
I20260620 20:22:24.485411 25199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:33377 every 8 connection(s)
I20260620 20:22:24.486164 25132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:24.488544 25200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:24.493101 25200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641: Bootstrap starting.
I20260620 20:22:24.494165 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25132
I20260620 20:22:24.494297 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:24.494572 2991 external_mini_cluster.cc:1428] Setting key 9fd396e1466ef28f7a2ac5eee4194817
I20260620 20:22:24.495729 25200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:24.496402 25200 log.cc:824] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:24.498196 25200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641: No bootstrap required, opened a new log
I20260620 20:22:24.498898 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:24.495468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41141 (local address 127.2.235.254:33377)
0620 20:22:24.495844 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:24.495856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:24.495885 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:22:24.496449 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:24.496462 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:24.496745 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:24.497045 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:24.497808 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:24.497814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:24.497834 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:24.497844 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:24.497888 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:24.497906 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:24.498015 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:24.498113 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 20:22:24.498263 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":118,"threads_started":1}
I20260620 20:22:24.501451 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:24.501961 25200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc8211ba30740b9800b76d79e5c9641" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33377 } }
I20260620 20:22:24.502157 25200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:24.502210 25200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc8211ba30740b9800b76d79e5c9641, State: Initialized, Role: FOLLOWER
I20260620 20:22:24.502590 25200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc8211ba30740b9800b76d79e5c9641" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33377 } }
I20260620 20:22:24.502750 25200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:24.502811 25200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:24.502897 25200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:24.503916 25200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc8211ba30740b9800b76d79e5c9641" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33377 } }
I20260620 20:22:24.504280 25200 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc8211ba30740b9800b76d79e5c9641; no voters:
I20260620 20:22:24.504609 25200 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:24.504665 25210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:24.504856 25210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc8211ba30740b9800b76d79e5c9641, State: Running, Role: LEADER
I20260620 20:22:24.505075 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:24.503452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37657 (local address 127.2.235.254:33377)
0620 20:22:24.503561 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:24.503566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:24.503731 (+ 165us) server_negotiation.cc:407] Connection header received
0620 20:22:24.503842 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:24.503847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:24.503877 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:24.503961 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:24.504623 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:24.504626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:24.504628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:24.504638 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:24.504668 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:24.504681 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:24.504753 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:24.504933 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 20:22:24.504983 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:22:24.505188 25210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc8211ba30740b9800b76d79e5c9641" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33377 } }
I20260620 20:22:24.505781 25200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:24.507118 25212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc8211ba30740b9800b76d79e5c9641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc8211ba30740b9800b76d79e5c9641" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33377 } } }
I20260620 20:22:24.507252 25212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:24.507562 25211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc8211ba30740b9800b76d79e5c9641. Latest consensus state: current_term: 1 leader_uuid: "1bc8211ba30740b9800b76d79e5c9641" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc8211ba30740b9800b76d79e5c9641" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 33377 } } }
I20260620 20:22:24.507637 25211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641 [sys.catalog]: This master's current role is: LEADER
W20260620 20:22:24.510151 25225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:24.510210 25225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:24.510263 25222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:24.510917 25222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:24.515596 25222 catalog_manager.cc:1347] Generated new cluster ID: 3817a8c0c0934bca87428af035f68094
I20260620 20:22:24.515688 25222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:24.530583 25222 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:24.530869 25222 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:24.535629 25222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc8211ba30740b9800b76d79e5c9641: Generated new TSK 0
I20260620 20:22:24.536168 25222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:24.541080 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25132
2026-06-20T20:22:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1303 ms)
[----------] 18 tests from AuthTokenIssuingTest (23496 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:24 dist-test-slave-8z53 krb5kdc[25231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:24 dist-test-slave-8z53 krb5kdc[25231](info): set up 2 sockets
Jun 20 20:22:24 dist-test-slave-8z53 krb5kdc[25231](info): commencing operation
krb5kdc: starting...
W20260620 20:22:25.573616 2991 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:25 dist-test-slave-8z53 krb5kdc[25231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:25.687572 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44029
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:44029,127.2.235.253:39739,127.2.235.252:45461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:25.788738 25244 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:22:25.788949 25244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:25.788970 25244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:25.789006 25244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:25.792461 25244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:25.792500 25244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:25.792515 25244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:25.792528 25244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:25.792541 25244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:25.796947 25244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:44029,127.2.235.253:39739,127.2.235.252:45461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:25.797724 25244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:25.798555 25244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:25.806026 25250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:25.806041 25252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:25.806051 25249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:25.806074 25244 server_base.cc:1034] running on GCE node
I20260620 20:22:25.806644 25244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:25.807539 25244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:25.808714 25244 hybrid_clock.cc:648] HybridClock initialized: now 1781986945808697 us; error 29 us; skew 500 ppm
Jun 20 20:22:25 dist-test-slave-8z53 krb5kdc[25231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:25.811101 25244 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:25.812048 25244 webserver.cc:460] Webserver started at http://127.2.235.254:38725/ using document root <none> and password file <none>
I20260620 20:22:25.812611 25244 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:25.812698 25244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:25.812898 25244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:25.814651 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "f1a785b0911a4d10b661608458b8e4cc"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-8z53"
server_key: "628dfafa899f82183aa5642eca843ca2"
server_key_iv: "82928e0bcaf4f7192ee6e1ec3755288e"
server_key_version: "encryptionkey@0"
I20260620 20:22:25.815156 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "f1a785b0911a4d10b661608458b8e4cc"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-8z53"
server_key: "628dfafa899f82183aa5642eca843ca2"
server_key_iv: "82928e0bcaf4f7192ee6e1ec3755288e"
server_key_version: "encryptionkey@0"
I20260620 20:22:25.818429 25244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:22:25.820544 25259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:25.821592 25244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:22:25.821722 25244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "f1a785b0911a4d10b661608458b8e4cc"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-8z53"
server_key: "628dfafa899f82183aa5642eca843ca2"
server_key_iv: "82928e0bcaf4f7192ee6e1ec3755288e"
server_key_version: "encryptionkey@0"
I20260620 20:22:25.821813 25244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:25.841189 25244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:25.843571 25244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:25.843742 25244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:25.849768 25244 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:44029
I20260620 20:22:25.850016 25311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:44029 every 8 connection(s)
I20260620 20:22:25.850544 25244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:25.852116 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25244
I20260620 20:22:25.852245 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:25.852481 2991 external_mini_cluster.cc:1428] Setting key 48a7d0d0a3b5a832108f4e04e0ae1688
I20260620 20:22:25.852953 25312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:25.856593 25312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.253@KRBTEST.COM" created.
Jun 20 20:22:25 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:25.871328 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.858578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44029 (local address 127.0.0.1:35412)
0620 20:22:25.858959 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.859001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:25.859735 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:25.860553 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:25.860558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:25.860969 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:25.861216 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:25.861228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.862569 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.862576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:25.863594 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:25.863603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.864403 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.864406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:25.864496 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:25.864996 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:25.865010 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:25.866435 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:25.868677 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:25.868685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:25.868692 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:25.868902 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:25.869232 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:25.869234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:25.869235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:25.869376 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:25.869954 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:25.869958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:25.870138 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 20:22:25.870315 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 20:22:25.870570 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":112,"threads_started":1}
I20260620 20:22:25.871333 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.858579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.2.235.254:44029)
0620 20:22:25.859058 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:25.859068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:25.859110 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:22:25.859998 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:25.860002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:25.860153 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:25.860320 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:25.861422 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.862449 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.863705 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.864263 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.864438 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:25.866597 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:25.866618 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:25.866631 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:25.866660 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:25.868556 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:25.869023 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:25.869028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:25.869030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:25.869143 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:25.869472 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:25.869477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:25.869482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:25.869857 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:25.869958 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:25.870435 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 20:22:25.870678 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":369,"thread_start_us":125,"threads_started":1}
I20260620 20:22:25.873451 25312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:25.874078 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.873857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47742)
0620 20:22:25.873952 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.874009 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:22:25.874265 25263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
I20260620 20:22:25.874986 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.874827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47758)
0620 20:22:25.874908 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.874945 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab.
Entry for principal kudu/127.2.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab.
W20260620 20:22:25.875152 25312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:39739: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
I20260620 20:22:25.875835 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:39739
--webserver_interface=127.2.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab
--principal=kudu/127.2.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:44029,127.2.235.253:39739,127.2.235.252:45461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:22:25.924369 25312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } attempt: 1
I20260620 20:22:25.925088 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.924846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47770)
0620 20:22:25.924978 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.925031 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:22:25.925896 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.925619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47776)
0620 20:22:25.925805 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.925848 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 20:22:25.926034 25312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:39739: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
W20260620 20:22:25.976853 25319 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:22:25.977087 25319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:25.977152 25319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:25.977228 25319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:25.981051 25319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:25.981135 25319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:25.981184 25319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:25.981222 25319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:25.981279 25319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:25.985909 25319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:44029,127.2.235.253:39739,127.2.235.252:45461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab
--principal=kudu/127.2.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:25.986824 25319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:25.987740 25319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:22:25.994172 25312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } attempt: 2
I20260620 20:22:25.995095 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.994704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47778)
0620 20:22:25.994922 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.995003 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 20:22:25.995728 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:25.995918 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:25.995981 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.995760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47786)
0620 20:22:25.995863 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.995912 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 20:22:25.996031 25319 server_base.cc:1034] running on GCE node
W20260620 20:22:25.996106 25312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:39739: Network error: Client connection negotiation failed: client connection to 127.2.235.253:39739: connect: Connection refused (error 111)
W20260620 20:22:25.995734 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:25.996678 25319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:25.997741 25319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:25.998924 25319 hybrid_clock.cc:648] HybridClock initialized: now 1781986945998891 us; error 41 us; skew 500 ppm
Jun 20 20:22:25 dist-test-slave-8z53 krb5kdc[25231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:26.001648 25319 init.cc:377] Logged in from keytab as kudu/127.2.235.253@KRBTEST.COM (short username kudu)
I20260620 20:22:26.002525 25319 webserver.cc:460] Webserver started at http://127.2.235.253:46311/ using document root <none> and password file <none>
I20260620 20:22:26.003067 25319 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:26.003132 25319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:26.003353 25319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:26.005155 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data/instance:
uuid: "70be368e12ef40ce946924a0584e7dd9"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-8z53"
server_key: "d50e3f1de532053849062d9c675e301c"
server_key_iv: "4e6065c503232886645ad2a32cd4b1e4"
server_key_version: "encryptionkey@0"
I20260620 20:22:26.005730 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal/instance:
uuid: "70be368e12ef40ce946924a0584e7dd9"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-8z53"
server_key: "d50e3f1de532053849062d9c675e301c"
server_key_iv: "4e6065c503232886645ad2a32cd4b1e4"
server_key_version: "encryptionkey@0"
I20260620 20:22:26.009334 25319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:26.011457 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:26.012530 25319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:26.012650 25319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal
uuid: "70be368e12ef40ce946924a0584e7dd9"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-8z53"
server_key: "d50e3f1de532053849062d9c675e301c"
server_key_iv: "4e6065c503232886645ad2a32cd4b1e4"
server_key_version: "encryptionkey@0"
I20260620 20:22:26.012743 25319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:26.034457 25319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:26.037243 25319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:26.037539 25319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:26.044269 25319 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.253:39739
I20260620 20:22:26.044535 25386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.253:39739 every 8 connection(s)
I20260620 20:22:26.045153 25319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
I20260620 20:22:26.047597 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:26.050853 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25319
I20260620 20:22:26.050972 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-1/wal/instance
I20260620 20:22:26.051259 2991 external_mini_cluster.cc:1428] Setting key ff241537cf182f12632c07b64d741a36
I20260620 20:22:26.051203 25387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.235.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.252@KRBTEST.COM" created.
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:26.066362 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.053117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.2.235.254:44029)
0620 20:22:26.053290 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.053295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.054047 (+ 752us) server_negotiation.cc:407] Connection header received
0620 20:22:26.054901 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.054907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.054971 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.055072 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.056093 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.056922 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.058155 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.058598 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.058716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.062358 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.062361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.062363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.062385 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.064377 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.064897 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.064901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.064902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.064980 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.065326 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.065329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.065331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.065503 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.065620 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.066083 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.066200 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:22:26.066992 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.053487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44029 (local address 127.0.0.1:35428)
0620 20:22:26.053913 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.053958 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.054705 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.055228 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.055233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.055533 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.055944 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.055955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.057052 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.057054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.058021 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.058031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.058741 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.058743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.058898 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.059516 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.059531 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.062176 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.064509 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.064514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.064527 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.064795 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.065069 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.065073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.065076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.065202 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.065628 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.065637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.065851 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.066011 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.066295 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
I20260620 20:22:26.068404 25387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab.
Entry for principal kudu/127.2.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab.
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:22:26.073791 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:45461
--webserver_interface=127.2.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab
--principal=kudu/127.2.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:44029,127.2.235.253:39739,127.2.235.252:45461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:22:26.077136 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.068837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47792)
0620 20:22:26.068946 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.068974 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.069046 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.069550 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.069552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.069711 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.069773 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.069778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.070797 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.070803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.071344 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.071352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.071913 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.071917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.072057 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.072425 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.072436 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.073660 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.075591 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.075593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.075595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.075854 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.076149 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.076151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.076152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.076218 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.076705 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.076707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.076771 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.076878 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.076986 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":25}
I20260620 20:22:26.077198 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.068938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.2.235.253:39739)
0620 20:22:26.069179 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.069193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.069210 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:22:26.069242 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.069284 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.069429 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.069569 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.069882 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.070689 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.071453 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.071800 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.071917 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.073806 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.073844 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.073859 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.073886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.075467 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.075960 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.075968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.075973 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.076060 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.076302 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.076308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.076319 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.076606 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.076737 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.076959 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.077065 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260620 20:22:26.079139 25387 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:26.079734 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.079527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35506)
0620 20:22:26.079623 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.079686 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:22:26.079933 25335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
I20260620 20:22:26.080727 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.080536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35522)
0620 20:22:26.080634 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.080681 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:22:26.080892 25387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:45461: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
I20260620 20:22:26.087255 25312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } attempt: 3
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:22:26.094857 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.087708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.2.235.253:39739)
0620 20:22:26.087824 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.087828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.087839 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:22:26.087951 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.087953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.087986 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.088049 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.088575 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.089006 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.090078 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.090294 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.090362 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.092235 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.092238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.092239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.092257 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.093529 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.093897 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.093902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.093903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.093967 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.094197 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.094201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.094203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.094337 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.094426 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.094645 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.094730 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:22:26.094866 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.087635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47808)
0620 20:22:26.087792 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.087804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.087871 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.088102 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.088105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.088303 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.088448 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.088456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.089094 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.089097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.089960 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.089970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.090386 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.090389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.090501 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.090927 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.090940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.092108 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.093627 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.093631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.093633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.093812 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.094045 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.094048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.094050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.094121 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.094419 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.094422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.094511 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.094602 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.094734 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 20:22:26.095893 25312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:26.096441 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.096239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35536)
0620 20:22:26.096339 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.096378 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 20:22:26.097078 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.096877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35542)
0620 20:22:26.097003 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.097039 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 20:22:26.097230 25312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:45461: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
I20260620 20:22:26.128423 25312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } attempt: 1
I20260620 20:22:26.129034 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.128805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35558)
0620 20:22:26.128920 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.128973 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:22:26.129740 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.129531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35570)
0620 20:22:26.129634 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.129684 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:22:26.129878 25312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:45461: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
I20260620 20:22:26.130085 25387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } attempt: 1
I20260620 20:22:26.130642 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.130422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35584)
0620 20:22:26.130524 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.130581 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:22:26.131222 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.131040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35592)
0620 20:22:26.131136 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.131180 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:22:26.131337 25387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:45461: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
W20260620 20:22:26.178476 25393 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:22:26.178743 25393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:26.178802 25393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:26.178880 25393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:26.182669 25393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:26.182742 25393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:26.182780 25393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:26.182824 25393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:26.182862 25393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:26.187852 25393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:44029,127.2.235.253:39739,127.2.235.252:45461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab
--principal=kudu/127.2.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:45461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:26.188776 25393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:26.189648 25393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:26.196583 25399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:26.196591 25402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:26.196700 25393 server_base.cc:1034] running on GCE node
W20260620 20:22:26.196592 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:26.197347 25393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:26.198324 25393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:26.199510 25393 hybrid_clock.cc:648] HybridClock initialized: now 1781986946199485 us; error 35 us; skew 500 ppm
I20260620 20:22:26.199481 25387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } attempt: 2
I20260620 20:22:26.200099 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.199860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35604)
0620 20:22:26.199979 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.200043 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:22:26.200747 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.200516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35606)
0620 20:22:26.200655 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.200696 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 20:22:26.200899 25387 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:45461: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:26.202296 25393 init.cc:377] Logged in from keytab as kudu/127.2.235.252@KRBTEST.COM (short username kudu)
I20260620 20:22:26.203166 25393 webserver.cc:460] Webserver started at http://127.2.235.252:33977/ using document root <none> and password file <none>
I20260620 20:22:26.203649 25393 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:26.203764 25393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:26.203999 25393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:26.205034 25312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } attempt: 2
I20260620 20:22:26.205653 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.205430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35616)
0620 20:22:26.205546 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.205594 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:22:26.205940 25393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data/instance:
uuid: "8644deb8aafd43b2b49477293a070d6b"
format_stamp: "Formatted at 2026-06-20 20:22:26 on dist-test-slave-8z53"
server_key: "04ede66df61d8e7113e6bf24cafcff96"
server_key_iv: "795faee465eaa37ba590929138819a02"
server_key_version: "encryptionkey@0"
I20260620 20:22:26.206357 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.206130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35630)
0620 20:22:26.206243 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.206310 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 20:22:26.206511 25312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:45461: Network error: Client connection negotiation failed: client connection to 127.2.235.252:45461: connect: Connection refused (error 111)
I20260620 20:22:26.206512 25393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal/instance:
uuid: "8644deb8aafd43b2b49477293a070d6b"
format_stamp: "Formatted at 2026-06-20 20:22:26 on dist-test-slave-8z53"
server_key: "04ede66df61d8e7113e6bf24cafcff96"
server_key_iv: "795faee465eaa37ba590929138819a02"
server_key_version: "encryptionkey@0"
I20260620 20:22:26.209774 25393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:26.211833 25409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:26.212828 25393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:26.212953 25393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal
uuid: "8644deb8aafd43b2b49477293a070d6b"
format_stamp: "Formatted at 2026-06-20 20:22:26 on dist-test-slave-8z53"
server_key: "04ede66df61d8e7113e6bf24cafcff96"
server_key_iv: "795faee465eaa37ba590929138819a02"
server_key_version: "encryptionkey@0"
I20260620 20:22:26.213047 25393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:26.226998 25393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:26.229628 25393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:26.229805 25393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:26.236100 25393 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.252:45461
I20260620 20:22:26.236369 25461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.252:45461 every 8 connection(s)
I20260620 20:22:26.236912 25393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
I20260620 20:22:26.239055 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25393
I20260620 20:22:26.239176 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986686246774-2991-0/minicluster-data/master-2/wal/instance
I20260620 20:22:26.239495 2991 external_mini_cluster.cc:1428] Setting key 2ec7cc47dc37a45b39cc950ee0d6d5bc
I20260620 20:22:26.239640 25462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:26.243978 25462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:26.253736 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.240684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.2.235.252:45461)
0620 20:22:26.241026 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.241040 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.241070 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:26.241662 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.241677 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.241947 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.242174 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.243042 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.243942 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.245095 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.245553 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.245722 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.248310 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.248321 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.248330 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.248354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.250235 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.250864 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.250869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.250874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.250995 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.251375 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.251378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.251379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.251845 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.251994 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.252654 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.252897 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
I20260620 20:22:26.255970 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.245685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.2.235.254:44029)
0620 20:22:26.245797 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.245802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.246128 (+ 326us) server_negotiation.cc:407] Connection header received
0620 20:22:26.246334 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.246337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.246392 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.246477 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.247128 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.247668 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.248597 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.248911 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.249026 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.251707 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.251711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.251752 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.251780 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.253871 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.254455 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.254458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.254460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.254552 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.254869 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.254875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.254876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.255134 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.255247 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.255689 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.255847 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:22:26.256194 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.245681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:44029 (local address 127.0.0.1:35438)
0620 20:22:26.246013 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.246054 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.246206 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.246523 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.246527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.246846 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.246996 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.247002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.247827 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.247831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.248452 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.248462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.249036 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.249040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.249135 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.249658 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.249670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.251550 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.253988 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.253997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.254002 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.254342 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.254651 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.254654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.254656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.254780 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.255270 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.255277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.255425 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.255580 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.255972 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":131,"threads_started":1}
I20260620 20:22:26.257164 25462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:22:26.267554 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.257632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:39739 (local address 127.0.0.1:47820)
0620 20:22:26.257917 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.257932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.258018 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.258266 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.258269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.258442 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.258534 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.258540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.259313 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.259318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.259966 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.259979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.260430 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.260434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.260538 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.260984 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.260998 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.262464 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.265973 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.265975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.265977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.266182 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.266481 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.266484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.266485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.266555 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.267055 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.267059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.267148 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.267259 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.267393 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:22:26.267597 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.256292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.2.235.254:44029)
0620 20:22:26.256648 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.256652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.256668 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:22:26.256759 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.256762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.256803 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.256880 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.257559 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.258286 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.259138 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.259366 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.259422 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.261959 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.261962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.261965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.261987 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.265782 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.266263 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.266267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.266269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.266346 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.266647 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.266650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.266652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.266859 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.267026 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.267343 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.267466 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260620 20:22:26.267604 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.257730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.2.235.253:39739)
0620 20:22:26.257856 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.257861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.258003 (+ 142us) server_negotiation.cc:407] Connection header received
0620 20:22:26.258109 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.258112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.258156 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.258230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.258663 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.259191 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.260071 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.260318 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.260431 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.262603 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.262606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.262609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.262633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.265851 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.266280 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.266283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.266284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.266376 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.266641 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.266644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.266646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.266910 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.267008 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.267324 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.267452 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:22:26.268862 25462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:22:26.277051 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.269268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35650)
0620 20:22:26.269540 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.269555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.269638 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.269826 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.269829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.269981 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.270054 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.270060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.270955 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.270959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.271675 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.271685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.272205 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.272209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.272287 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.272688 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.272714 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.273942 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.275691 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.275696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.275700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.275957 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.276211 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.276216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.276220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.276302 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.276601 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.276604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.276691 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.276796 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.276903 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260620 20:22:26.277168 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.269386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.2.235.252:45461)
0620 20:22:26.269559 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.269562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.269599 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:22:26.269721 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.269723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.269749 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.269801 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.270162 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.270837 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.271793 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.272100 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.272187 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.274068 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.274073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.274076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.274100 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.275567 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.276047 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.276050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.276051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.276128 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.276383 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.276385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.276386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.276512 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.276576 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.276866 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.276965 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:22:26.278002 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.268991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.2.235.253:39739)
0620 20:22:26.269098 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.269101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.269165 (+ 64us) server_negotiation.cc:407] Connection header received
0620 20:22:26.269312 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.269314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.269354 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.269444 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.270163 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.270835 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.271793 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.272094 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.272187 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.274568 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.274572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.274574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.274597 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.276317 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.276710 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.276712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.276713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.276802 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.277093 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.277099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.277101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.277334 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.277557 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.277692 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.277818 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 20:22:26.280583 25462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: Bootstrap starting.
I20260620 20:22:26.281032 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:26.282940 25462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:26.282383 25286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f1a785b0911a4d10b661608458b8e4cc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35446
I20260620 20:22:26.283643 25462 log.cc:824] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:26.285392 25462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: No bootstrap required, opened a new log
I20260620 20:22:26.287998 25462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.288178 25462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8644deb8aafd43b2b49477293a070d6b, State: Initialized, Role: FOLLOWER
I20260620 20:22:26.288484 25462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.288901 25468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.289014 25468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.289640 25462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:26.291970 25387 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } attempt: 3
W20260620 20:22:26.293887 25479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:26.294010 25479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:26.292719 25393 master_runner.cc:186] Error getting master registration for 127.2.235.254:44029: OK, instance_id { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" instance_seqno: 1781986945848135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:22:26.295619 25393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
I20260620 20:22:26.300055 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.292282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35664)
0620 20:22:26.292427 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.292440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.292512 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.292715 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.292718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.292874 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.292949 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.292956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.293724 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.293727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.294511 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.294521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.295060 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.295075 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.295151 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.295664 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.295679 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.296956 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.298692 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.298694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.298696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.298882 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.299164 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.299168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.299170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.299252 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.299590 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.299593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.299684 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.299787 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.299915 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 20:22:26.300055 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.292327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.2.235.252:45461)
0620 20:22:26.292409 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.292413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.292479 (+ 66us) server_negotiation.cc:407] Connection header received
0620 20:22:26.292589 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.292591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.292621 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.292703 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.293042 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.293610 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.294635 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.294955 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.295053 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.297078 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.297081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.297083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.297103 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.298558 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.298975 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.298979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.298981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.299077 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.299342 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.299346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.299347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.299498 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.299597 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.299835 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.299926 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 20:22:26.303200 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: Bootstrap starting.
I20260620 20:22:26.305675 25387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:26.305665 25312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } attempt: 3
I20260620 20:22:26.306440 25387 log.cc:824] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:26.308354 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: No bootstrap required, opened a new log
Jun 20 20:22:26 dist-test-slave-8z53 krb5kdc[25231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
I20260620 20:22:26.311946 25387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.312137 25387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70be368e12ef40ce946924a0584e7dd9, State: Initialized, Role: FOLLOWER
I20260620 20:22:26.312417 25387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.312851 25482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.312988 25482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.313548 25387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:26.314090 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.306052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.2.235.252:45461)
0620 20:22:26.306140 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.306143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.306198 (+ 55us) server_negotiation.cc:407] Connection header received
0620 20:22:26.306302 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.306306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.306341 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.306438 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.306874 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.307626 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.308371 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.308703 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.308793 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.310793 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.310819 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.310823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.310844 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.312421 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.312902 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.312905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.312907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.312985 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.313359 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.313362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.313364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.313507 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.313608 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.313886 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.313974 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 20:22:26.314090 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.305999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:45461 (local address 127.0.0.1:35666)
0620 20:22:26.306145 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:26.306158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:26.306235 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:26.306460 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:26.306462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:26.306637 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:26.306709 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.306715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.307739 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.307742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.308251 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:26.308259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.308804 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.308808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:26.308902 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.309337 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:26.309352 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:26.310662 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:26.312516 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.312519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.312521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.312755 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.313153 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:26.313155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:26.313171 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:26.313275 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:26.313651 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:26.313654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:26.313737 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 20:22:26.313838 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:22:26.313954 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:22:26.317440 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: Bootstrap starting.
W20260620 20:22:26.318854 25493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:26.318969 25493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:26.316572 25319 master_runner.cc:186] Error getting master registration for 127.2.235.254:44029: OK, instance_id { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" instance_seqno: 1781986945848135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:22:26.319666 25319 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:22:26.319748 25312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:26.320391 25312 log.cc:824] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:26.322216 25312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: No bootstrap required, opened a new log
I20260620 20:22:26.324884 25312 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.325062 25312 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a785b0911a4d10b661608458b8e4cc, State: Initialized, Role: FOLLOWER
I20260620 20:22:26.325366 25312 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.325834 25496 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.326064 25496 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.326422 25312 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:26.326479 25286 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:22:26.326587 25286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:26.327754 25286 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.328672 25286 leader_election.cc:288] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [CANDIDATE]: Term 1 election: Requested vote from peers 70be368e12ef40ce946924a0584e7dd9 (127.2.235.253:39739), 8644deb8aafd43b2b49477293a070d6b (127.2.235.252:45461)
I20260620 20:22:26.329149 25361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1a785b0911a4d10b661608458b8e4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70be368e12ef40ce946924a0584e7dd9"
I20260620 20:22:26.329412 25361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:26.329717 25436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1a785b0911a4d10b661608458b8e4cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8644deb8aafd43b2b49477293a070d6b"
I20260620 20:22:26.329996 25436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:26.330715 25361 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a785b0911a4d10b661608458b8e4cc in term 1.
I20260620 20:22:26.331230 25436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a785b0911a4d10b661608458b8e4cc in term 1.
I20260620 20:22:26.331094 25263 leader_election.cc:302] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70be368e12ef40ce946924a0584e7dd9, f1a785b0911a4d10b661608458b8e4cc; no voters:
I20260620 20:22:26.332753 25496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:26.333338 25496 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [term 1 LEADER]: Becoming Leader. State: Replica: f1a785b0911a4d10b661608458b8e4cc, State: Running, Role: LEADER
I20260620 20:22:26.333626 25496 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } }
I20260620 20:22:26.335198 25496 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a785b0911a4d10b661608458b8e4cc. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.335311 25496 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: This master's current role is: LEADER
W20260620 20:22:26.335196 25507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:26.337630 25507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:26.338639 25515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:26.339442 25515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 20:22:26.344992 25265 tablet.cc:2369] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:26.348680 25436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [term 1 FOLLOWER]: Refusing update from remote peer f1a785b0911a4d10b661608458b8e4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:26.349344 25361 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [term 1 FOLLOWER]: Refusing update from remote peer f1a785b0911a4d10b661608458b8e4cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:26.350010 25496 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:26.350374 25496 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:26.353956 25482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a785b0911a4d10b661608458b8e4cc. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.354067 25482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.356108 25468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a785b0911a4d10b661608458b8e4cc. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.356227 25468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.357707 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.338734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.2.235.252:45461)
0620 20:22:26.338835 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.338839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.345559 (+ 6720us) server_negotiation.cc:407] Connection header received
0620 20:22:26.345702 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.345705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.345744 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.345794 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.346650 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.347302 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.351070 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.351422 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.351497 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.353143 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.353148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.353150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.353173 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.355792 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.356281 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.356285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.356287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.356376 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.356674 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.356677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.356678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.356888 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.356983 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.357389 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.357512 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 20:22:26.360536 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.338282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.2.235.253:39739)
0620 20:22:26.338447 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.338451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.345316 (+ 6865us) server_negotiation.cc:407] Connection header received
0620 20:22:26.345492 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.345495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.345546 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.345594 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.346535 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.347274 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.348880 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.349184 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.349267 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.354201 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.354205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.354210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.354233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.355774 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.359294 (+ 3520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.359297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.359299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.359390 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.359688 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.359691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.359693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.359871 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.359973 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.360269 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.360382 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 20:22:26.361114 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:26.337652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.2.235.254:44029)
0620 20:22:26.337756 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.337760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.344399 (+ 6639us) server_negotiation.cc:407] Connection header received
0620 20:22:26.344561 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.344565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.344602 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.344645 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.345637 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.346372 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.355863 (+ 9491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:26.356178 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:26.356248 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:26.357890 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:26.357893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:26.357896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:26.357921 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:26.359505 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.359944 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.359947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.359948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.360022 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:26.360271 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:26.360274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:26.360276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:26.360453 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:26.360543 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:26.360857 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:22:26.360967 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:22:26.365620 25496 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.365731 25496 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:26.366729 25520 mvcc.cc:205] Tried to move back new op lower bound from 7299018532223422464 to 7299018532185448448. Current Snapshot: MvccSnapshot[applied={T|T < 7299018532223422464}]
I20260620 20:22:26.366830 25482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.366936 25482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.369496 25496 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.369593 25496 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:26.370332 25468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1a785b0911a4d10b661608458b8e4cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a785b0911a4d10b661608458b8e4cc" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 44029 } } peers { permanent_uuid: "70be368e12ef40ce946924a0584e7dd9" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 39739 } } peers { permanent_uuid: "8644deb8aafd43b2b49477293a070d6b" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 45461 } } }
I20260620 20:22:26.370435 25468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:26.370251 25522 mvcc.cc:205] Tried to move back new op lower bound from 7299018532223422464 to 7299018532185448448. Current Snapshot: MvccSnapshot[applied={T|T < 7299018532223422464}]
I20260620 20:22:26.370656 25519 mvcc.cc:205] Tried to move back new op lower bound from 7299018532223422464 to 7299018532185448448. Current Snapshot: MvccSnapshot[applied={T|T < 7299018532223422464}]
I20260620 20:22:26.371876 25515 catalog_manager.cc:1347] Generated new cluster ID: 37713d1e45fc48458815652f116c9631
I20260620 20:22:26.371933 25515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:26.385002 25515 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:26.386073 25515 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:26.399880 25515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1a785b0911a4d10b661608458b8e4cc: Generated new TSK 0
I20260620 20:22:26.400527 25515 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:22:26.426627 25316 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.424307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.2.235.254:44029)
0620 20:22:26.424417 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.424422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.425652 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 20:22:26.425864 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.425867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.425910 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.425955 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.426517 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35464: BlockingRecv error: recv got EOF from 127.0.0.1:35464 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 20:22:26.427855 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.425001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47848 (local address 127.2.235.253:39739)
0620 20:22:26.425118 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.425122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.426969 (+ 1847us) server_negotiation.cc:407] Connection header received
0620 20:22:26.427134 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.427138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.427178 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.427223 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.427737 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47848: BlockingRecv error: recv got EOF from 127.0.0.1:47848 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 20:22:26.429704 25465 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.428578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.2.235.252:45461)
0620 20:22:26.428710 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.428714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.428917 (+ 203us) server_negotiation.cc:407] Connection header received
0620 20:22:26.429068 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.429071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.429110 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.429156 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.429608 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35680: BlockingRecv error: recv got EOF from 127.0.0.1:35680 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 20:22:26.433130 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.432054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.2.235.253:39739)
0620 20:22:26.432147 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.432151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.432338 (+ 187us) server_negotiation.cc:407] Connection header received
0620 20:22:26.432509 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.432512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.432585 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.432628 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.433053 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47862: BlockingRecv error: recv got EOF from 127.0.0.1:47862 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:22:26.437647 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.436312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.2.235.253:39739)
0620 20:22:26.436394 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.436398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.436845 (+ 447us) server_negotiation.cc:407] Connection header received
0620 20:22:26.436995 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.436997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.437042 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.437082 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.437571 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47874: BlockingRecv error: recv got EOF from 127.0.0.1:47874 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 20:22:26.442795 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.441404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.2.235.253:39739)
0620 20:22:26.441489 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.441492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.441975 (+ 483us) server_negotiation.cc:407] Connection header received
0620 20:22:26.442140 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.442144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.442185 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.442230 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.442703 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47888: BlockingRecv error: recv got EOF from 127.0.0.1:47888 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 20:22:26.450033 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.448511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.2.235.253:39739)
0620 20:22:26.448651 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.448655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.449153 (+ 498us) server_negotiation.cc:407] Connection header received
0620 20:22:26.449418 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.449422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.449464 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.449509 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.449956 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47890: BlockingRecv error: recv got EOF from 127.0.0.1:47890 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 20:22:26.461112 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.459732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.2.235.253:39739)
0620 20:22:26.459829 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.459833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.460306 (+ 473us) server_negotiation.cc:407] Connection header received
0620 20:22:26.460490 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.460493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.460533 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.460578 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.461036 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47900: BlockingRecv error: recv got EOF from 127.0.0.1:47900 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 20:22:26.480645 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.479029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.2.235.253:39739)
0620 20:22:26.479130 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.479135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.479645 (+ 510us) server_negotiation.cc:407] Connection header received
0620 20:22:26.479870 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.479874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.479936 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.479996 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.480548 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47912: BlockingRecv error: recv got EOF from 127.0.0.1:47912 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 20:22:26.516754 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.515534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.2.235.253:39739)
0620 20:22:26.515748 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.515754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.515967 (+ 213us) server_negotiation.cc:407] Connection header received
0620 20:22:26.516030 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.516033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.516084 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.516187 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.516634 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47926: BlockingRecv error: recv got EOF from 127.0.0.1:47926 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 20:22:26.584582 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.583482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.2.235.253:39739)
0620 20:22:26.583622 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.583627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.583759 (+ 132us) server_negotiation.cc:407] Connection header received
0620 20:22:26.583893 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.583895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.583941 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.584012 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.584437 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33284: BlockingRecv error: recv got EOF from 127.0.0.1:33284 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 20:22:26.716069 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.714800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.2.235.253:39739)
0620 20:22:26.714932 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.714937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.715168 (+ 231us) server_negotiation.cc:407] Connection header received
0620 20:22:26.715315 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.715317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.715367 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.715471 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.715902 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33288: BlockingRecv error: recv got EOF from 127.0.0.1:33288 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 20:22:26.975142 25392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:26.974091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.2.235.253:39739)
0620 20:22:26.974213 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:26.974218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:26.974313 (+ 95us) server_negotiation.cc:407] Connection header received
0620 20:22:26.974505 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:26.974508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:26.974557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:26.974634 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:26.975041 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33302: BlockingRecv error: recv got EOF from 127.0.0.1:33302 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:22:27.295194 25479 catalog_manager.cc:1259] Loaded cluster ID: 37713d1e45fc48458815652f116c9631
I20260620 20:22:27.295261 25479 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: loading cluster ID for follower catalog manager: success
I20260620 20:22:27.296909 25479 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: acquiring CA information for follower catalog manager: success
I20260620 20:22:27.298192 25479 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8644deb8aafd43b2b49477293a070d6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:22:27.320045 25493 catalog_manager.cc:1259] Loaded cluster ID: 37713d1e45fc48458815652f116c9631
I20260620 20:22:27.320102 25493 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: loading cluster ID for follower catalog manager: success
I20260620 20:22:27.321718 25493 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: acquiring CA information for follower catalog manager: success
I20260620 20:22:27.322747 25493 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70be368e12ef40ce946924a0584e7dd9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:22:27.492641 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.489144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.2.235.253:39739)
0620 20:22:27.489386 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.489390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.489407 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:22:27.489545 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.489547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.489664 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.489754 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:27.489962 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.490556 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.491305 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.491511 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.491610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.491864 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:27.492275 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:27.492355 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.492403 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.492454 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260620 20:22:27.498624 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.495399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.2.235.252:45461)
0620 20:22:27.495604 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.495608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.495620 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:22:27.495688 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.495690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.495798 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.495877 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:27.496040 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.496505 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.497184 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.497474 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.497571 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.497810 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:27.498255 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:27.498344 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.498394 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.498437 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260620 20:22:27.504781 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.500981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.2.235.254:44029)
0620 20:22:27.501332 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.501337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.501352 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:22:27.501435 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.501437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.501526 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.501651 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:27.501902 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.502525 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.503521 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.503762 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.503869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.504143 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:27.504393 (+ 250us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:27.504482 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.504593 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.504649 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20260620 20:22:27.505188 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.501269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.2.235.253:39739)
0620 20:22:27.501403 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.501407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.501921 (+ 514us) server_negotiation.cc:407] Connection header received
0620 20:22:27.502075 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.502080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.502120 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.502204 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:27.502422 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.502920 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.504044 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.504303 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.504355 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.504720 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:27.504790 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:27.504858 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.505025 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.505072 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:22:27.505776 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.501306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.2.235.252:45461)
0620 20:22:27.501402 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.501406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.501797 (+ 391us) server_negotiation.cc:407] Connection header received
0620 20:22:27.501910 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.501913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.501968 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.502039 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:27.502454 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.503125 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.504161 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.504456 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.504580 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.505054 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:27.505144 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:27.505195 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.505621 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.505662 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 20:22:27.507750 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25244
I20260620 20:22:27.513782 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25319
I20260620 20:22:27.519721 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25393
2026-06-20T20:22:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2979 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:27 dist-test-slave-8z53 krb5kdc[25611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:27 dist-test-slave-8z53 krb5kdc[25611](info): set up 2 sockets
Jun 20 20:22:27 dist-test-slave-8z53 krb5kdc[25611](info): commencing operation
krb5kdc: starting...
W20260620 20:22:28.493474 2991 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/test-user.keytab.
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.235.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
Entry for principal kudu/127.2.235.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab.
I20260620 20:22:28.595305 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41469
--webserver_interface=127.2.235.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41469,127.2.235.253:44943,127.2.235.252:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:28.694964 25624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:22:28.695225 25624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:28.695294 25624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:28.695366 25624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:28.699015 25624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:28.699083 25624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:28.699117 25624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:28.699170 25624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:28.699206 25624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:28.703589 25624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41469,127.2.235.253:44943,127.2.235.252:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.254.keytab
--principal=kudu/127.2.235.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.254:41469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.235.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:28.704396 25624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:28.705369 25624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:28.712870 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:28.713001 25624 server_base.cc:1034] running on GCE node
W20260620 20:22:28.712908 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:28.712874 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:28.713610 25624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:28.714478 25624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:28.715662 25624 hybrid_clock.cc:648] HybridClock initialized: now 1781986948715624 us; error 40 us; skew 500 ppm
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:28.722752 25624 init.cc:377] Logged in from keytab as kudu/127.2.235.254@KRBTEST.COM (short username kudu)
I20260620 20:22:28.723465 25624 webserver.cc:460] Webserver started at http://127.2.235.254:39417/ using document root <none> and password file <none>
I20260620 20:22:28.723923 25624 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:28.724004 25624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:28.724197 25624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:28.725912 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data/instance:
uuid: "53a1a42d571f46c4bb8957bbfeef8699"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-8z53"
server_key: "44293b18b802901fc7f8ee3a2e166303"
server_key_iv: "9da4045868cc50ec7cbd2ba60707a134"
server_key_version: "encryptionkey@0"
I20260620 20:22:28.726437 25624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal/instance:
uuid: "53a1a42d571f46c4bb8957bbfeef8699"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-8z53"
server_key: "44293b18b802901fc7f8ee3a2e166303"
server_key_iv: "9da4045868cc50ec7cbd2ba60707a134"
server_key_version: "encryptionkey@0"
I20260620 20:22:28.729755 25624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:28.732016 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:28.733119 25624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:28.733256 25624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal
uuid: "53a1a42d571f46c4bb8957bbfeef8699"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-8z53"
server_key: "44293b18b802901fc7f8ee3a2e166303"
server_key_iv: "9da4045868cc50ec7cbd2ba60707a134"
server_key_version: "encryptionkey@0"
I20260620 20:22:28.733429 25624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:28.747208 25624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:28.749936 25624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:28.750113 25624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:28.756480 25624 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.254:41469
I20260620 20:22:28.756742 25691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.254:41469 every 8 connection(s)
I20260620 20:22:28.757337 25624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:28.759709 25692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:28.759966 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25624
I20260620 20:22:28.760107 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-0/wal/instance
I20260620 20:22:28.760350 2991 external_mini_cluster.cc:1428] Setting key 6e0311329228ba35edd2c410043c4929
I20260620 20:22:28.763065 25692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.235.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.253@KRBTEST.COM" created.
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:28.777123 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.766102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.2.235.254:41469)
0620 20:22:28.766377 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:28.766387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:28.766427 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:22:28.766752 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:28.766756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:28.766935 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:28.767113 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:28.767929 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.768745 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.769829 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.770238 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.770495 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.772817 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:28.772823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:28.772840 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:28.772859 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:28.774362 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:28.774852 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:28.774859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:28.774861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:28.774955 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:28.775274 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:28.775277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:28.775279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:28.775589 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:28.775721 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:28.776072 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 20:22:28.776281 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260620 20:22:28.777123 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.765128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41469 (local address 127.0.0.1:58752)
0620 20:22:28.765626 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.765662 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:28.766480 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:28.767113 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:28.767118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:28.767511 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:28.767768 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.767792 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.768884 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.768887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.769713 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.769723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.770379 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.770383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.770558 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.771086 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:28.771100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:28.772700 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:28.774481 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:28.774490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:28.774499 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:28.774729 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:28.775051 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:28.775054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:28.775056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:28.775162 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:28.775697 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:28.775702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:28.775836 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 20:22:28.775984 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 20:22:28.776327 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":161,"thread_start_us":170,"threads_started":1}
I20260620 20:22:28.779361 25692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:28.779920 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.779678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35672)
0620 20:22:28.779787 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.779870 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:22:28.780117 25641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
I20260620 20:22:28.780917 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.780718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35688)
0620 20:22:28.780828 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.780873 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:22:28.781038 25692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:44943: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab.
Entry for principal kudu/127.2.235.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab.
I20260620 20:22:28.783727 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:44943
--webserver_interface=127.2.235.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab
--principal=kudu/127.2.235.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41469,127.2.235.253:44943,127.2.235.252:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:22:28.830277 25692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } attempt: 1
I20260620 20:22:28.831135 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.830835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35696)
0620 20:22:28.830992 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.831061 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:22:28.831915 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.831698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35702)
0620 20:22:28.831798 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.831849 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:22:28.832048 25692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:44943: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
W20260620 20:22:28.886435 25699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:22:28.886677 25699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:28.886735 25699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:28.886804 25699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:28.890753 25699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:28.890802 25699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:28.890821 25699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:28.890849 25699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:28.890862 25699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:28.895442 25699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41469,127.2.235.253:44943,127.2.235.252:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.253.keytab
--principal=kudu/127.2.235.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.253:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.235.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:28.896386 25699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:28.897331 25699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:22:28.900218 25692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } attempt: 2
I20260620 20:22:28.900976 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.900694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35704)
0620 20:22:28.900828 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.900895 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:22:28.901949 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.901690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35712)
0620 20:22:28.901823 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.901880 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 20:22:28.902118 25692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.253:44943: Network error: Client connection negotiation failed: client connection to 127.2.235.253:44943: connect: Connection refused (error 111)
W20260620 20:22:28.905681 25704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:28.905695 25705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:28.905791 25699 server_base.cc:1034] running on GCE node
W20260620 20:22:28.905659 25707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:28.906387 25699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:28.907290 25699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:28.908465 25699 hybrid_clock.cc:648] HybridClock initialized: now 1781986948908432 us; error 42 us; skew 500 ppm
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:28.911314 25699 init.cc:377] Logged in from keytab as kudu/127.2.235.253@KRBTEST.COM (short username kudu)
I20260620 20:22:28.912067 25699 webserver.cc:460] Webserver started at http://127.2.235.253:46227/ using document root <none> and password file <none>
I20260620 20:22:28.912750 25699 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:28.912848 25699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:28.913038 25699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:28.914836 25699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data/instance:
uuid: "13b08921f0564420be95d477f01964e0"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-8z53"
server_key: "3e5f00180e16f7f0fa795fa90c703bac"
server_key_iv: "638c599ff2803885f7e6c5a467923636"
server_key_version: "encryptionkey@0"
I20260620 20:22:28.915472 25699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal/instance:
uuid: "13b08921f0564420be95d477f01964e0"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-8z53"
server_key: "3e5f00180e16f7f0fa795fa90c703bac"
server_key_iv: "638c599ff2803885f7e6c5a467923636"
server_key_version: "encryptionkey@0"
I20260620 20:22:28.920055 25699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 20:22:28.925236 25714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:28.926621 25699 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 20:22:28.926806 25699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal
uuid: "13b08921f0564420be95d477f01964e0"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-8z53"
server_key: "3e5f00180e16f7f0fa795fa90c703bac"
server_key_iv: "638c599ff2803885f7e6c5a467923636"
server_key_version: "encryptionkey@0"
I20260620 20:22:28.926923 25699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:28.953847 25699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:28.956861 25699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:28.957068 25699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:28.963874 25699 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.253:44943
I20260620 20:22:28.964121 25766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.253:44943 every 8 connection(s)
I20260620 20:22:28.964746 25699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/data/info.pb
I20260620 20:22:28.967150 25767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:28.968962 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25699
I20260620 20:22:28.969090 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-1/wal/instance
I20260620 20:22:28.969385 2991 external_mini_cluster.cc:1428] Setting key 14752a32243cdddad0537583265a1186
I20260620 20:22:28.971583 25767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.235.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.235.252@KRBTEST.COM" created.
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:28.985528 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.973372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.2.235.254:41469)
0620 20:22:28.973504 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:28.973508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:28.974264 (+ 756us) server_negotiation.cc:407] Connection header received
0620 20:22:28.975144 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:28.975150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:28.975215 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:28.975313 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:28.976415 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.977288 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.978545 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.978779 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.978913 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.981654 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:28.981657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:28.981659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:28.981684 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:28.983503 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:28.983982 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:28.983988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:28.983993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:28.984083 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:28.984394 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:28.984399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:28.984403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:28.984645 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:28.984758 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:28.985212 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 20:22:28.985345 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:22:28.986191 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.973723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41469 (local address 127.0.0.1:58766)
0620 20:22:28.974112 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.974156 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:28.974946 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:28.975489 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:28.975496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:28.975838 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:28.976256 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.976270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.977407 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.977410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.978405 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.978416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.978982 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.978993 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.979136 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.979754 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:28.979776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:28.981450 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:28.983623 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:28.983627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:28.983637 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:28.983880 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:28.984179 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:28.984181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:28.984183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:28.984300 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:28.984754 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:28.984758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:28.984973 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 20:22:28.985114 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 20:22:28.985409 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260620 20:22:28.987474 25767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.235.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab.
Entry for principal kudu/127.2.235.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab.
I20260620 20:22:28.993305 25692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } attempt: 3
I20260620 20:22:28.994082 2991 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9vOW7R/build/debug/bin/kudu
/tmp/dist-test-task9vOW7R/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:34935
--webserver_interface=127.2.235.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab
--principal=kudu/127.2.235.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.235.212:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.235.254:41469,127.2.235.253:44943,127.2.235.252:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:22:28.997157 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.987982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35722)
0620 20:22:28.988066 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.988081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:28.988147 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:28.988676 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:28.988679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:28.988861 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:28.988951 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.988955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.990012 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.990018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.990621 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.990629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.991214 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.991217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.991313 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.991717 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:28.991744 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:28.992942 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:28.995739 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:28.995744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:28.995747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:28.995999 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:28.996261 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:28.996264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:28.996266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:28.996356 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:28.996718 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:28.996720 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:28.996798 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 20:22:28.996899 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:22:28.997007 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 20:22:28.997875 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.988062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.2.235.253:44943)
0620 20:22:28.988340 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:28.988348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:28.988364 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:22:28.988409 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:28.988413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:28.988562 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:28.988694 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:28.989052 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.989846 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.990735 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.991105 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.991218 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.993054 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:28.993069 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:28.993079 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:28.993097 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:28.995600 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:28.996106 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:28.996110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:28.996111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:28.996174 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:28.996435 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:28.996437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:28.996438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:28.996613 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:28.996745 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:28.996977 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 20:22:28.997081 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
Jun 20 20:22:28 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:22:28.999548 25767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:29.000180 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.999911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43140)
0620 20:22:29.000029 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.000105 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:22:29.000375 25718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
I20260620 20:22:29.001168 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.000977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43146)
0620 20:22:29.001051 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.001089 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 20:22:29.001334 25767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:34935: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
I20260620 20:22:29.002586 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.993916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.2.235.253:44943)
0620 20:22:28.994218 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:28.994228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:28.994244 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:22:28.994290 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:28.994295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:28.994342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:28.994430 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:28.994819 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.995356 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.996682 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.997086 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.997229 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.999289 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:28.999299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:28.999301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:28.999324 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.000885 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.001488 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.001491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.001493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.001568 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.001861 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.001863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.001864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.001985 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.002047 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.002401 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.002489 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":103,"threads_started":1}
I20260620 20:22:29.002874 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:28.993689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35724)
0620 20:22:28.993812 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:28.993837 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:28.993909 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:28.994465 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:28.994469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:28.994625 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:28.994719 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.994723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.995454 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.995458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.996583 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:28.996590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:28.997227 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:28.997231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:28.997366 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:28.997802 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:28.997822 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:28.999155 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.000974 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.000979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.000983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.001354 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.001666 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.001672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.001675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.001766 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.002070 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.002092 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.002199 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.002330 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.002466 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:22:29.003512 25692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:29.004104 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.003916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43156)
0620 20:22:29.004016 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.004056 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:22:29.004805 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.004595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43162)
0620 20:22:29.004717 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.004753 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 20:22:29.004990 25692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:34935: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
I20260620 20:22:29.036160 25692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } attempt: 1
I20260620 20:22:29.036777 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.036546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43164)
0620 20:22:29.036669 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.036718 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:22:29.037423 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.037200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43178)
0620 20:22:29.037335 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.037382 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:22:29.037544 25692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:34935: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
I20260620 20:22:29.050539 25767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } attempt: 1
I20260620 20:22:29.051134 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.050927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43184)
0620 20:22:29.051032 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.051077 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:22:29.051800 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.051582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43196)
0620 20:22:29.051695 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.051744 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:22:29.051927 25767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:34935: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
W20260620 20:22:29.096155 25774 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:22:29.096413 25774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:29.096479 25774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:29.096551 25774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:29.100283 25774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:29.100355 25774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:29.100390 25774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:29.100442 25774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:29.100477 25774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:29.105144 25774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.235.212:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.235.254:41469,127.2.235.253:44943,127.2.235.252:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/krb5kdc/kudu_127.2.235.252.keytab
--principal=kudu/127.2.235.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.235.252:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.235.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:22:29.106062 25774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:29.106951 25774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:22:29.112720 25692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } attempt: 2
I20260620 20:22:29.113445 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.113177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43210)
0620 20:22:29.113304 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.113366 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 20:22:29.114162 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.113959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43218)
0620 20:22:29.114048 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.114098 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 20:22:29.114313 25692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:34935: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
W20260620 20:22:29.114539 25780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:29.114548 25781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:29.114667 25774 server_base.cc:1034] running on GCE node
W20260620 20:22:29.114660 25783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:29.115147 25774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:29.116058 25774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:29.117215 25774 hybrid_clock.cc:648] HybridClock initialized: now 1781986949117190 us; error 34 us; skew 500 ppm
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:29.119731 25774 init.cc:377] Logged in from keytab as kudu/127.2.235.252@KRBTEST.COM (short username kudu)
I20260620 20:22:29.120079 25767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } attempt: 2
I20260620 20:22:29.120512 25774 webserver.cc:460] Webserver started at http://127.2.235.252:42125/ using document root <none> and password file <none>
I20260620 20:22:29.120723 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.120482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43222)
0620 20:22:29.120607 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.120661 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 20:22:29.121129 25774 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:29.121210 25774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:29.121449 25774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:29.121483 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.121210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43236)
0620 20:22:29.121373 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.121422 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 20:22:29.121611 25767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.235.252:34935: Network error: Client connection negotiation failed: client connection to 127.2.235.252:34935: connect: Connection refused (error 111)
I20260620 20:22:29.123208 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data/instance:
uuid: "2167d57ce4e449fb8f5b4c21b2868694"
format_stamp: "Formatted at 2026-06-20 20:22:29 on dist-test-slave-8z53"
server_key: "9d31cb3d0a79610f33327090bff4d7f4"
server_key_iv: "7cfaa52751db88852276fa2778ffc37b"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.123715 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal/instance:
uuid: "2167d57ce4e449fb8f5b4c21b2868694"
format_stamp: "Formatted at 2026-06-20 20:22:29 on dist-test-slave-8z53"
server_key: "9d31cb3d0a79610f33327090bff4d7f4"
server_key_iv: "7cfaa52751db88852276fa2778ffc37b"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.127032 25774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:22:29.129142 25790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:29.130348 25774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:29.130493 25774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data,/tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal
uuid: "2167d57ce4e449fb8f5b4c21b2868694"
format_stamp: "Formatted at 2026-06-20 20:22:29 on dist-test-slave-8z53"
server_key: "9d31cb3d0a79610f33327090bff4d7f4"
server_key_iv: "7cfaa52751db88852276fa2778ffc37b"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.130611 25774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:29.141414 25774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:29.144483 25774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:29.144693 25774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:29.151503 25774 rpc_server.cc:307] RPC server started. Bound to: 127.2.235.252:34935
I20260620 20:22:29.151777 25842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.235.252:34935 every 8 connection(s)
I20260620 20:22:29.152369 25774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/data/info.pb
I20260620 20:22:29.154940 25843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:29.158536 25843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:29.160178 2991 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9vOW7R/build/debug/bin/kudu as pid 25774
I20260620 20:22:29.160343 2991 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9vOW7R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986686246774-2991-0/minicluster-data/master-2/wal/instance
I20260620 20:22:29.160652 2991 external_mini_cluster.cc:1428] Setting key b71be11720534b2519185aba95defdde
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
I20260620 20:22:29.171089 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.160180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.2.235.254:41469)
0620 20:22:29.160315 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.160321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.161042 (+ 721us) server_negotiation.cc:407] Connection header received
0620 20:22:29.162097 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.162100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.162143 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.162205 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.163131 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.163673 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.164706 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.165049 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.165157 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.167465 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.167468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.167470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.167494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.169064 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.169603 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.169606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.169608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.169685 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.170034 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.170037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.170038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.170216 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.170340 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.170813 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.170920 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:22:29.171840 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.160498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.254:41469 (local address 127.0.0.1:58768)
0620 20:22:29.160890 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.160937 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:29.161896 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:29.162267 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:29.162275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:29.162671 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:29.162974 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.162987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.163780 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.163784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.164597 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.164605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.165197 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.165201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.165410 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.165976 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:29.165990 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:29.167323 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.169167 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.169178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.169188 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.169513 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.169787 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.169790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.169792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.169937 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.170317 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.170324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.170577 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.170708 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.170994 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260620 20:22:29.172241 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.161271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.2.235.252:34935)
0620 20:22:29.161589 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.161598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.161808 (+ 210us) server_negotiation.cc:407] Connection header received
0620 20:22:29.161988 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.162006 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.162234 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.162384 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.163175 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.164140 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.165153 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.165653 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.165734 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.168449 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.168457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.168471 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.168499 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.170232 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.170828 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.170833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.170834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.170912 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.171313 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.171316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.171317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.171552 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.171727 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.171932 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.172058 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":228,"thread_start_us":131,"threads_started":1}
I20260620 20:22:29.173014 25843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.254@KRBTEST.COM
I20260620 20:22:29.181412 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.173460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.253:44943 (local address 127.0.0.1:35732)
0620 20:22:29.173541 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.173554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:29.173621 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:29.174006 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:29.174009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:29.174179 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:29.174289 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.174296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.175134 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.175137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.175741 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.175751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.176191 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.176195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.176299 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.176752 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:29.176772 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:29.177941 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.179877 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.179882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.179885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.180131 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.180472 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.180478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.180481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.180574 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.180992 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.180994 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.181059 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.181129 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.181209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 20:22:29.181586 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.173562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.2.235.253:44943)
0620 20:22:29.173774 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.173779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.173794 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:22:29.173840 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.173843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.173889 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.173992 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.174415 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.175006 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.175865 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.176088 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.176160 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.178081 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.178089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.178091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.178111 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.179772 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.180229 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.180234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.180240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.180358 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.180668 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.180672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.180674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.180890 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.180992 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.181214 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.181383 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 20:22:29.182204 25843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:22:29.182988 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.174267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.2.235.254:41469)
0620 20:22:29.174365 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.174371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.174385 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:22:29.174485 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.174488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.174526 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.174599 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.175116 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.175559 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.176570 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.176804 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.176918 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.179480 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.179485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.179487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.179518 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.181306 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.181781 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.181785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.181787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.181870 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.182170 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.182173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.182175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.182405 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.182622 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.182776 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.182891 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.252@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.235.253@KRBTEST.COM
I20260620 20:22:29.190687 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.182620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43260)
0620 20:22:29.183075 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.183091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:29.183165 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:29.183372 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:29.183376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:29.183544 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:29.183624 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.183630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.184564 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.184566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.185088 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.185095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.185592 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.185595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.185704 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.186101 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:29.186111 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:29.187185 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.189090 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.189095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.189097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.189412 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.189729 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.189735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.189740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.189855 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.190336 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.190339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.190412 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.190497 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.190585 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":5}
I20260620 20:22:29.190876 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.182690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.2.235.252:34935)
0620 20:22:29.182792 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.182796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.183227 (+ 431us) server_negotiation.cc:407] Connection header received
0620 20:22:29.183251 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.183253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.183281 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.183337 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.183718 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.184458 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.185200 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.185489 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.185595 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.187335 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.187338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.187340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.187361 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.188979 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.189514 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.189520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.189524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.189611 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.189954 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.189961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.189965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.190207 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.190341 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.190581 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.190693 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":30}
I20260620 20:22:29.192663 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.184008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.2.235.253:44943)
0620 20:22:29.184099 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.184103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.184117 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:22:29.184173 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.184175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.184205 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.184266 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.184887 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.185398 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.186186 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.186393 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.186486 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.189213 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.189216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.189217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.189238 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.190998 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.191493 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.191496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.191498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.191592 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.191873 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.191876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.191878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.192094 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.192311 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.192390 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.192495 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 20:22:29.194275 25843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: Bootstrap starting.
I20260620 20:22:29.195945 2991 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:29.196550 25843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:29.197232 25843 log.cc:824] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:29.197082 25666 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "53a1a42d571f46c4bb8957bbfeef8699"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58774
I20260620 20:22:29.199106 25843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: No bootstrap required, opened a new log
I20260620 20:22:29.201853 25843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.202070 25843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2167d57ce4e449fb8f5b4c21b2868694, State: Initialized, Role: FOLLOWER
I20260620 20:22:29.202359 25843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.202844 25849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.202980 25849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.203595 25843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:22:29.208218 25860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:29.208309 25860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:29.207247 25774 master_runner.cc:186] Error getting master registration for 127.2.235.254:41469: OK, instance_id { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" instance_seqno: 1781986948754668 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:22:29.210531 25774 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:22:29.212788 25767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } attempt: 3
I20260620 20:22:29.213461 25692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } attempt: 3
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.253@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
Jun 20 20:22:29 dist-test-slave-8z53 krb5kdc[25611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.235.254@KRBTEST.COM for kudu/127.2.235.252@KRBTEST.COM
I20260620 20:22:29.221874 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.213111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43262)
0620 20:22:29.213210 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.213226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:29.213326 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:29.213608 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:29.213611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:29.213785 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:29.213877 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.213883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.214814 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.214817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.215564 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.215572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.216074 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.216077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.216154 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.216614 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:29.216627 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:29.217984 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.220211 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.220216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.220219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.220496 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.220811 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.220814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.220816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.220906 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.221416 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.221419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.221507 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.221605 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.221724 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 20:22:29.222414 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.213180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.2.235.252:34935)
0620 20:22:29.213294 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.213298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.213309 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:22:29.213428 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.213430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.213483 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.213582 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.213989 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.214708 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.215677 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.215981 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.216062 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.218114 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.218118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.218121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.218145 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.220109 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.220611 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.220614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.220616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.220698 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.221007 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.221010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.221012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.221226 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.221383 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.222163 (+ 780us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.222265 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:22:29.223615 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.213803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.235.252:34935 (local address 127.0.0.1:43266)
0620 20:22:29.213897 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.213922 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:29.213988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:29.214328 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:29.214330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:29.214485 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:29.214553 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.214557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.215554 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.215558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.216363 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.216374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.217519 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.217523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.217611 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.218416 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:29.218430 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:29.219865 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.222066 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.222070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.222072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.222316 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.222632 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.222635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.222637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.222725 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.223134 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.223137 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.223237 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.223343 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.223469 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 20:22:29.223677 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.213878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.2.235.252:34935)
0620 20:22:29.214113 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.214120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.214134 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:22:29.214162 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.214165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.214200 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.214270 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.214674 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.215428 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.216519 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.217403 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.217502 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.220002 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.220005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.220007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.220029 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.221918 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.222433 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.222436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.222437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.222525 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.222835 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.222837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.222839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.223033 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.223142 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.223402 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.223502 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260620 20:22:29.225193 25767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: Bootstrap starting.
I20260620 20:22:29.226117 25692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: Bootstrap starting.
I20260620 20:22:29.227555 25767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:29.227931 25692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:29.228376 25767 log.cc:824] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:29.228605 25692 log.cc:824] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:29.230346 25767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: No bootstrap required, opened a new log
I20260620 20:22:29.230546 25692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: No bootstrap required, opened a new log
I20260620 20:22:29.233194 25692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.233234 25767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.233489 25692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a1a42d571f46c4bb8957bbfeef8699, State: Initialized, Role: FOLLOWER
I20260620 20:22:29.233601 25767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b08921f0564420be95d477f01964e0, State: Initialized, Role: FOLLOWER
I20260620 20:22:29.233779 25692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.233979 25767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.234294 25865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.234679 25865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.234946 25692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:29.234978 25666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:22:29.235071 25666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:29.235313 25767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:29.236274 25666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.237895 25817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53a1a42d571f46c4bb8957bbfeef8699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2167d57ce4e449fb8f5b4c21b2868694"
I20260620 20:22:29.238128 25817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:29.239369 25817 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a1a42d571f46c4bb8957bbfeef8699 in term 1.
I20260620 20:22:29.240711 25741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53a1a42d571f46c4bb8957bbfeef8699" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13b08921f0564420be95d477f01964e0"
I20260620 20:22:29.240926 25741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:29.237257 25666 leader_election.cc:288] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [CANDIDATE]: Term 1 election: Requested vote from peers 13b08921f0564420be95d477f01964e0 (127.2.235.253:44943), 2167d57ce4e449fb8f5b4c21b2868694 (127.2.235.252:34935)
I20260620 20:22:29.242125 25741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a1a42d571f46c4bb8957bbfeef8699 in term 1.
I20260620 20:22:29.240140 25643 leader_election.cc:302] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2167d57ce4e449fb8f5b4c21b2868694, 53a1a42d571f46c4bb8957bbfeef8699; no voters:
W20260620 20:22:29.244284 25888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:29.244361 25888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:29.246893 25866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.247008 25866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.248430 25624 master_runner.cc:263] No leader master found from master 53a1a42d571f46c4bb8957bbfeef8699
I20260620 20:22:29.248485 25624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:22:29.249681 25882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:29.249750 25882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:29.250016 25865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:29.250392 25865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [term 1 LEADER]: Becoming Leader. State: Replica: 53a1a42d571f46c4bb8957bbfeef8699, State: Running, Role: LEADER
I20260620 20:22:29.251415 25699 master_runner.cc:263] No leader master found from master 13b08921f0564420be95d477f01964e0
I20260620 20:22:29.251464 25699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:22:29.251381 25645 tablet.cc:2369] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:29.252100 25865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } }
I20260620 20:22:29.253837 25865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a1a42d571f46c4bb8957bbfeef8699. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.253961 25865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:29.256412 25900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:29.257351 25900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:29.257958 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.245686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.2.235.254:41469)
0620 20:22:29.245780 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.245784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.249492 (+ 3708us) server_negotiation.cc:407] Connection header received
0620 20:22:29.249673 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.249678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.249727 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.249770 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.250579 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.251203 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.252153 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.252434 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.252511 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.254208 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.254212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.254216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.254239 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.255838 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.256594 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.256597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.256599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.256689 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.256992 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.256995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.256996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.257175 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.257319 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.257712 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.257834 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:22:29.262276 25741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [term 1 FOLLOWER]: Refusing update from remote peer 53a1a42d571f46c4bb8957bbfeef8699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:29.262277 25817 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [term 1 FOLLOWER]: Refusing update from remote peer 53a1a42d571f46c4bb8957bbfeef8699: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:29.262908 25865 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:29.263132 25898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:29.266359 25849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a1a42d571f46c4bb8957bbfeef8699. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.266353 25866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a1a42d571f46c4bb8957bbfeef8699. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.266451 25849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.266451 25866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.268344 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.244788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.2.235.253:44943)
0620 20:22:29.244904 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.244910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.248776 (+ 3866us) server_negotiation.cc:407] Connection header received
0620 20:22:29.248953 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.248956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.248993 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.249035 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.250024 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.250745 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.255272 (+ 4527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.255596 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.255660 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.258989 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.258993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.258995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.259020 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.261013 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.266968 (+ 5955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.266973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.266976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.267064 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.267367 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.267370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.267373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.267571 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.267715 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.268075 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.268206 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:22:29.269598 25865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.269697 25865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:29.269837 25901 mvcc.cc:205] Tried to move back new op lower bound from 7299018544172683264 to 7299018544140083200. Current Snapshot: MvccSnapshot[applied={T|T < 7299018544172683264}]
I20260620 20:22:29.270823 25849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.271186 25849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.272784 25903 mvcc.cc:205] Tried to move back new op lower bound from 7299018544172683264 to 7299018544140083200. Current Snapshot: MvccSnapshot[applied={T|T < 7299018544172683264}]
I20260620 20:22:29.273840 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.246328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.2.235.252:34935)
0620 20:22:29.246428 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.246432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.252054 (+ 5622us) server_negotiation.cc:407] Connection header received
0620 20:22:29.254494 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.254497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.254533 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.254571 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.256076 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.256814 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.259436 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.259757 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.259822 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.270648 (+ 10826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.270651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.270654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.270675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.272215 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.272685 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.272688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.272689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.272761 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.273014 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.273017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.273019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.273187 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.273330 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.273602 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.273700 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:22:29.275480 25898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.275611 25898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:29.276557 25866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53a1a42d571f46c4bb8957bbfeef8699" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53a1a42d571f46c4bb8957bbfeef8699" member_type: VOTER last_known_addr { host: "127.2.235.254" port: 41469 } } peers { permanent_uuid: "13b08921f0564420be95d477f01964e0" member_type: VOTER last_known_addr { host: "127.2.235.253" port: 44943 } } peers { permanent_uuid: "2167d57ce4e449fb8f5b4c21b2868694" member_type: VOTER last_known_addr { host: "127.2.235.252" port: 34935 } } }
I20260620 20:22:29.276660 25866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:29.278420 25900 catalog_manager.cc:1347] Generated new cluster ID: a21ddec5a77f4ddebe9b57a19e4cdc6c
I20260620 20:22:29.278477 25900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:29.291934 25900 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:29.292902 25900 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:29.308990 25900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53a1a42d571f46c4bb8957bbfeef8699: Generated new TSK 0
I20260620 20:22:29.309641 25900 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:22:29.347065 25862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.344160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.2.235.252:34935)
0620 20:22:29.344282 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.344287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.346070 (+ 1783us) server_negotiation.cc:407] Connection header received
0620 20:22:29.346242 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.346246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.346292 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.346338 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.346945 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43284: BlockingRecv error: recv got EOF from 127.0.0.1:43284 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 20:22:29.348631 25696 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.344907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.2.235.254:41469)
0620 20:22:29.345016 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.345020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.347501 (+ 2481us) server_negotiation.cc:407] Connection header received
0620 20:22:29.347963 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.347967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.348014 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.348062 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.348518 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58784: BlockingRecv error: recv got EOF from 127.0.0.1:58784 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:22:29.349843 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.345570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.2.235.253:44943)
0620 20:22:29.345692 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.345696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.347705 (+ 2009us) server_negotiation.cc:407] Connection header received
0620 20:22:29.348863 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.348867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.348921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.348982 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.349720 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35770: BlockingRecv error: recv got EOF from 127.0.0.1:35770 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 20:22:29.353885 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.352581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.2.235.253:44943)
0620 20:22:29.352686 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.352690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.352967 (+ 277us) server_negotiation.cc:407] Connection header received
0620 20:22:29.353170 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.353173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.353237 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.353353 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.353771 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35778: BlockingRecv error: recv got EOF from 127.0.0.1:35778 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 20:22:29.358660 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.357328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.2.235.253:44943)
0620 20:22:29.357411 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.357415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.357897 (+ 482us) server_negotiation.cc:407] Connection header received
0620 20:22:29.358060 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.358063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.358103 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.358148 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.358581 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35794: BlockingRecv error: recv got EOF from 127.0.0.1:35794 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 20:22:29.363607 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.362420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.2.235.253:44943)
0620 20:22:29.362497 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.362500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.362913 (+ 413us) server_negotiation.cc:407] Connection header received
0620 20:22:29.363059 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.363062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.363098 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.363138 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.363537 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35804: BlockingRecv error: recv got EOF from 127.0.0.1:35804 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 20:22:29.370922 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.369493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.2.235.253:44943)
0620 20:22:29.369602 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.369605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.370048 (+ 443us) server_negotiation.cc:407] Connection header received
0620 20:22:29.370205 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.370208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.370246 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.370290 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.370809 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35812: BlockingRecv error: recv got EOF from 127.0.0.1:35812 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 20:22:29.382030 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.380717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.2.235.253:44943)
0620 20:22:29.380812 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.380816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.381267 (+ 451us) server_negotiation.cc:407] Connection header received
0620 20:22:29.381452 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.381454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.381493 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.381534 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.381955 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35824: BlockingRecv error: recv got EOF from 127.0.0.1:35824 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 20:22:29.401556 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.400092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.2.235.253:44943)
0620 20:22:29.400194 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.400199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.400677 (+ 478us) server_negotiation.cc:407] Connection header received
0620 20:22:29.400870 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.400873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.400918 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.400966 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.401469 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35838: BlockingRecv error: recv got EOF from 127.0.0.1:35838 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 20:22:29.437065 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.435933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.2.235.253:44943)
0620 20:22:29.436047 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.436051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.436257 (+ 206us) server_negotiation.cc:407] Connection header received
0620 20:22:29.436395 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.436399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.436450 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.436529 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.436953 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35850: BlockingRecv error: recv got EOF from 127.0.0.1:35850 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 20:22:29.503870 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.502909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.2.235.253:44943)
0620 20:22:29.503018 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.503022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.503165 (+ 143us) server_negotiation.cc:407] Connection header received
0620 20:22:29.503264 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.503266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.503309 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.503380 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.503725 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35852: BlockingRecv error: recv got EOF from 127.0.0.1:35852 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 20:22:29.635051 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.633854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.2.235.253:44943)
0620 20:22:29.633989 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.633993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.634197 (+ 204us) server_negotiation.cc:407] Connection header received
0620 20:22:29.634316 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.634318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.634364 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.634435 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.634895 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35858: BlockingRecv error: recv got EOF from 127.0.0.1:35858 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 20:22:29.894305 25773 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:29.892971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.2.235.253:44943)
0620 20:22:29.893211 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.893217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.893346 (+ 129us) server_negotiation.cc:407] Connection header received
0620 20:22:29.893518 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.893522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.893578 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.893670 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.894188 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35860: BlockingRecv error: recv got EOF from 127.0.0.1:35860 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
I20260620 20:22:30.209486 25860 catalog_manager.cc:1259] Loaded cluster ID: a21ddec5a77f4ddebe9b57a19e4cdc6c
I20260620 20:22:30.209559 25860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: loading cluster ID for follower catalog manager: success
I20260620 20:22:30.211198 25860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: acquiring CA information for follower catalog manager: success
I20260620 20:22:30.212306 25860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2167d57ce4e449fb8f5b4c21b2868694: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:22:30.245402 25888 catalog_manager.cc:1259] Loaded cluster ID: a21ddec5a77f4ddebe9b57a19e4cdc6c
I20260620 20:22:30.245452 25888 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: loading cluster ID for follower catalog manager: success
I20260620 20:22:30.246932 25888 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: acquiring CA information for follower catalog manager: success
I20260620 20:22:30.247967 25888 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13b08921f0564420be95d477f01964e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:22:30.411964 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.408259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.2.235.253:44943)
0620 20:22:30.408502 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.408506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.408570 (+ 64us) server_negotiation.cc:407] Connection header received
0620 20:22:30.408696 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.408699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.408836 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.408931 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:30.409177 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.409714 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.410521 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.410727 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.410814 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.411092 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:30.411497 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:30.411584 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.411705 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.411750 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":60,"threads_started":1}
I20260620 20:22:30.418493 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.414901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.2.235.252:34935)
0620 20:22:30.415132 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.415136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.415163 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:22:30.415301 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.415304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.415412 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.415540 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:30.415772 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.416276 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.417089 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.417356 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.417466 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.417748 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:30.418102 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:30.418226 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.418288 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.418346 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":63,"threads_started":1}
I20260620 20:22:30.425814 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.421618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.2.235.252:34935)
0620 20:22:30.421718 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.421722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.421956 (+ 234us) server_negotiation.cc:407] Connection header received
0620 20:22:30.422043 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.422045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.422087 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.422161 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:30.422415 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.423169 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.424472 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.424793 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.424865 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.425286 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:30.425375 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:30.425438 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.425610 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.425667 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:22:30.425920 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.421307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.2.235.253:44943)
0620 20:22:30.421408 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.421412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.421597 (+ 185us) server_negotiation.cc:407] Connection header received
0620 20:22:30.421785 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.421788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.421846 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.421885 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:30.422290 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.423120 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.424919 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.425114 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.425220 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.425487 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:30.425567 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:30.425638 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.425747 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.425799 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 20:22:30.426584 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.421077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.2.235.254:41469)
0620 20:22:30.421452 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.421457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.421471 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:22:30.421700 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.421704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.421869 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.421979 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:30.423315 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.423817 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.424723 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.425156 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.425266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.425652 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:30.426013 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:30.426114 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.426355 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.426415 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":57,"threads_started":1}
I20260620 20:22:30.429602 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25624
I20260620 20:22:30.436302 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25699
I20260620 20:22:30.442232 2991 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9vOW7R/build/debug/bin/kudu with pid 25774
2026-06-20T20:22:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2922 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5902 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (264190 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST